CN113177965A - Coal rock full-component extraction method based on improved U-net network and application thereof - Google Patents
Coal rock full-component extraction method based on improved U-net network and application thereof Download PDFInfo
- Publication number
- CN113177965A CN113177965A CN202110381605.XA CN202110381605A CN113177965A CN 113177965 A CN113177965 A CN 113177965A CN 202110381605 A CN202110381605 A CN 202110381605A CN 113177965 A CN113177965 A CN 113177965A
- Authority
- CN
- China
- Prior art keywords
- coal rock
- improved
- image
- coal
- network model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 239000003245 coal Substances 0.000 title claims abstract description 143
- 239000011435 rock Substances 0.000 title claims abstract description 136
- 238000000605 extraction Methods 0.000 title claims abstract description 36
- 238000000034 method Methods 0.000 claims abstract description 46
- 230000004913 activation Effects 0.000 claims abstract description 29
- 238000009826 distribution Methods 0.000 claims abstract description 20
- 238000012805 post-processing Methods 0.000 claims abstract description 16
- 238000012545 processing Methods 0.000 claims abstract description 15
- 238000005457 optimization Methods 0.000 claims abstract description 10
- 230000006872 improvement Effects 0.000 claims abstract description 8
- 230000006870 function Effects 0.000 claims description 40
- 238000012549 training Methods 0.000 claims description 36
- 230000008569 process Effects 0.000 claims description 22
- 238000012795 verification Methods 0.000 claims description 10
- 230000015654 memory Effects 0.000 claims description 8
- 238000002372 labelling Methods 0.000 claims description 6
- 238000013519 translation Methods 0.000 claims description 6
- 230000000903 blocking effect Effects 0.000 claims description 5
- 238000012216 screening Methods 0.000 claims description 5
- 238000007781 pre-processing Methods 0.000 claims description 4
- 238000002360 preparation method Methods 0.000 claims description 2
- 230000011218 segmentation Effects 0.000 abstract description 29
- 238000003709 image segmentation Methods 0.000 abstract description 16
- 230000007547 defect Effects 0.000 abstract description 4
- 238000005070 sampling Methods 0.000 description 33
- 239000011347 resin Substances 0.000 description 14
- 229920005989 resin Polymers 0.000 description 14
- 238000004364 calculation method Methods 0.000 description 11
- 238000013528 artificial neural network Methods 0.000 description 8
- 239000002245 particle Substances 0.000 description 7
- 239000000853 adhesive Substances 0.000 description 6
- 230000001070 adhesive effect Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 6
- 238000010606 normalization Methods 0.000 description 6
- 238000011176 pooling Methods 0.000 description 6
- 238000004458 analytical method Methods 0.000 description 5
- 230000008901 benefit Effects 0.000 description 5
- 238000013135 deep learning Methods 0.000 description 5
- 238000013527 convolutional neural network Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 235000011837 pasties Nutrition 0.000 description 4
- 238000002790 cross-validation Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 230000018109 developmental process Effects 0.000 description 3
- 230000002708 enhancing effect Effects 0.000 description 3
- 238000011156 evaluation Methods 0.000 description 3
- 238000002474 experimental method Methods 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 230000009466 transformation Effects 0.000 description 3
- 229920002101 Chitin Polymers 0.000 description 2
- 239000006185 dispersion Substances 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 239000012634 fragment Substances 0.000 description 2
- 238000011423 initialization method Methods 0.000 description 2
- 230000000873 masking effect Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 239000000126 substance Substances 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- NAWXUBYGYWOOIX-SFHVURJKSA-N (2s)-2-[[4-[2-(2,4-diaminoquinazolin-6-yl)ethyl]benzoyl]amino]-4-methylidenepentanedioic acid Chemical compound C1=CC2=NC(N)=NC(N)=C2C=C1CCC1=CC=C(C(=O)N[C@@H](CC(=C)C(O)=O)C(O)=O)C=C1 NAWXUBYGYWOOIX-SFHVURJKSA-N 0.000 description 1
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000003321 amplification Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 239000000446 fuel Substances 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 238000003064 k means clustering Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000013138 pruning Methods 0.000 description 1
- 238000003908 quality control method Methods 0.000 description 1
- 239000002994 raw material Substances 0.000 description 1
- 230000001172 regenerating effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000002759 z-score normalization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/60—Rotation of a whole image or part thereof
-
- G06T5/92—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10056—Microscopic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a coal rock total component extraction method based on an improved U-net network and application thereof, wherein the method comprises the following steps: the method comprises the steps of obtaining an image to be processed, inputting the image to be processed into an improved Unet network model, outputting a coal rock component category probability distribution map by the improved Unet network model, carrying out post-processing on the coal rock component category probability distribution map to obtain a coal rock full-component image, wherein the post-processing comprises carrying out optimization processing by using a full-connection condition random field; compared with the Unet network model, the improved Unet network model has the improvement that the Mish activation function is adopted to replace the original Relu activation function. The method overcomes the defects that the existing traditional image segmentation algorithm has poor adaptability, cannot deal with complex coal rock microscopic images and needs manual intervention when performing coal rock image segmentation; which enables efficient segmentation of images; the method has good adaptability, does not need manual intervention, and has great application prospect.
Description
Technical Field
The invention belongs to the technical field of coal rock analysis, relates to a method for completing coal rock analysis by applying an image segmentation technology, and particularly relates to a coal rock total component extraction method based on an improved U-net network and application thereof.
Background
Coal is one of important energy sources, and can be mainly used as a power fuel and a raw material of metallurgical industry. Due to the non-regenerative property of the coal resources, the improvement of the utilization rate of the coal resources has very important significance. Since the 80 s of the 20 th century, with the vigorous development of science and technology, coal petrography has rapidly developed. Due to the wide application of new technologies and new methods, coal petrography research is increasingly developed from macro to micro and supermicro, from qualitative to quantitative, and from single research to multiple technology cross research. The development of automated systems for coal quality control is achieved primarily by observing under a microscope the differences between the different major microscopic components present in the polished coal block. Given that manual lithofacies analysis of coal requires skilled operators and the results obtained can be highly subjective, an ideal approach to solving these problems is to employ automated image analysis.
The segmentation of the coal rock microscopic image is a precondition and guarantee for coal rock automatic analysis, and before specific classification and extraction of each component, denoising pretreatment such as background segmentation is required to be carried out on the coal rock microscopic image, so that interference factors influencing grouping identification of the microscopic component are removed, and an effective coal particle image area is extracted.
Image segmentation is an important technique for extracting an object of interest from a background in image preprocessing. Existing image segmentation techniques can be divided into the following categories: threshold segmentation, boundary-based segmentation, region-based segmentation, cluster-based segmentation, and hybrid segmentation. With the great improvement of the performance of a computer and the continuous optimization of a traditional image processing algorithm, a plurality of coal rock microscopic images are preprocessed by adopting methods such as an adaptive threshold (such as OTSU), watershed segmentation, k-means clustering and the like, and mixed methods such as region growing and the like to remove background resin, but the problem of poor adaptability of the preprocessing methods in background segmentation is still not effectively solved, and a great amount of improvement and more manual intervention are needed in the application process to improve the background segmentation effect.
Therefore, the development of the coal rock total-component extraction method which has good adaptability and does not need manual intervention has practical significance.
Disclosure of Invention
The invention aims to overcome the defects that the existing traditional image segmentation algorithm has low adaptability, cannot deal with complex coal rock microscopic images and needs manual intervention when performing coal rock image segmentation, and provides a coal rock full-component extraction method which has good adaptability and does not need manual intervention.
The invention uses deep learning method to extract image characteristic, which is different from traditional image processing method, and can automatically extract characteristic from image without designing manual extraction rule, and Convolutional Neural Networks (CNNs) are widely used as an efficient characteristic extraction structure. The Full Convolution Network (FCN) is a special application of CNN, and its main idea is to use CNN as a powerful feature extractor, and replace the full connection layer with convolution to output a spatial feature map instead of a classification score, which has been proven to be an ideal way to complete the task of image segmentation. U-Net is further improved on the basis of FCN, and is successfully applied to biomedical image segmentation, and the performance of the U-Net is excellent in the biomedical image segmentation. The U-Net structure combines shallow information with deep information using skip connections to achieve accurate pixel-level positioning. Specifically, the invention provides a coal rock total component extraction method based on a semantic segmentation network (U-Net network) to further improve the accuracy and robustness of extracting coal rock microscopic total components, and the method solves two difficulties in the coal rock total component extraction process by using an improved U-Net network: 1. dividing the resin adhesive and the shell group in the gray overlapping area; 2. eliminating paste-like and semitransparent bonding formed by the falling of coal particles under the polished surface.
In order to achieve the purpose, the invention provides the following technical scheme:
the coal rock full-component extraction method based on the improved U-net network comprises the steps of obtaining an image to be processed, inputting the image into an improved Unet network model, outputting a coal rock component category probability distribution map by the improved Unet network model, and carrying out post-processing on the coal rock component category probability distribution map to obtain a coal rock full-component image, wherein the post-processing comprises carrying out optimization processing by using a fully-connected conditional random field (Dense CRF);
compared with the Unet network model, the improved Unet network model has the improvement that the Mish activation function is adopted to replace the original Relu activation function;
the training process of the improved Unet network model is a process of continuously adjusting model parameters by taking images of a training data set as input and taking a coal rock component category distribution graph corresponding to the input as theoretical output, wherein the termination condition of training is that the verification precision of the improved Unet network model is continuously stopped lifting by N (20) rounds when a verification sample model is used for verification, all the images in the training data set are manually calibrated, and the training data set comprises the coal rock component category distribution graphs corresponding to all the images.
The invention relates to a coal rock total component extraction method, which utilizes an improved Unet network to complete coal rock total component extraction, a U-Net structure combines shallow information and deep information by using skip connection so as to realize accurate pixel level positioning, and simultaneously improves a traditional U-Net structure, adopts a Mish activation function to replace an original Relu activation function, further inhibits the gradient dispersion problem of the deep network by combining a residual structure, and uses a fully-connected conditional random field (Dense CRF) to perform optimization processing in the post-processing process, wherein the method is an improvement mode of the conditional random field, can process classification results obtained by model inference by combining the relation among all pixels in an original image, optimizes rough and uncertain marks in the classification image, corrects finely-broken fault regions and optimizes segmentation boundaries. The method can realize effective segmentation of the image (can segment the resin adhesive and the shell group in the gray overlapping area and can eliminate the pasty and semitransparent bonding formed by the falling of coal particles under a polished surface), and has wide application prospect.
As a preferred technical scheme:
compared with the Unet network model, the improved Unet network model has the improvement that the residual volume block in Resnet is adopted to replace the original volume block. Namely, the residual structure in the resnet is used to enhance the capability of extracting the features of the network, and the gradient dispersion problem of the deep network is effectively solved by matching the identity mapping in the residual structure with the Mish activation function (as described above, the Mish activation function is used to replace the Relu activation function to transform the residual volume block). In addition, in the aspect of model branch reduction, the precision of the model is ensured, and a down-sampling layer is reduced, so that the parameter quantity of the model is effectively reduced.
According to the coal rock total component extraction method based on the improved U-net network, the training data set is obtained through the following process:
(1) acquiring and preprocessing a coal rock microscopic image;
(2) acquiring a training data set:
(2.1) marking the coal rock microscopic image obtained in the step (1) to finish data set marking;
(2.2) screening a typical sample preparation data set, wherein the data set comprises a normal sample (coal rock components and background resin are clearly distinguished) and a difficult sample (a sample with a resin adhesive overlapped with a shell component or a pasty and semitransparent bonding phenomenon formed by coal particles falling off from a polished surface);
and (2.3) performing data expansion on the data set obtained in the step (2.2).
The coal rock total component extraction method based on the improved U-net network comprises the following specific steps of (1):
(1.1) using a Zeiss Axio-Zoom camera to shoot a coal slide to obtain an original coal rock microscopic image;
(1.2) carrying out overlapping blocking processing on the original coal rock microscopic image to obtain a subgraph with the size of 1024 multiplied by 1024, wherein the overlapping area is 1/n of the size of the subgraph.
The coal rock total component extraction method based on the improved U-net network comprises the following specific operations:
manually calibrating the coal rock microscopic image obtained in the step (1), wherein all coal rock component areas in the microscopic image are foreground, and non-coal rock component areas are background (including non-coal rock component areas such as high-brightness areas caused by mushy, semitransparent bonding and overexposure);
the specific operation of the data expansion is as follows:
and (3) randomly zooming, translating, overturning, rotating and contrast enhancing the coal rock microscopic image labeled in the data set obtained in the step (2.2) to expand the data set, wherein the enhancing operations of translating, overturning and rotating which relate to position transformation need to perform synchronous operation on the label graph of each image.
According to the coal rock total component extraction method based on the improved U-net network, the training process of the improved Unet network model is as follows:
(1) data online enhancement and standardization processing;
the data expansion in the invention comprises two parts, the data expansion is offline data expansion, the data expansion used in the step is online data expansion in the training process, the generalization and stability of the network model are improved by combining the two parts, and the online data expansion specifically comprises: random mirror image turning, random gamma transformation, random brightness enhancement, random contrast enhancement, random rotation, random translation and the like;
the normalization process is specifically z-score normalization, which operates as:
and calculating the Mean value of all coal rock microscopic images in the data set as Mean and the standard deviation as Std. By means of the pixel value I of each pixel in the coal rock microscopic imagexyzNormalization was performed as follows.
(2) The construction of the improved Unet network model is as follows:
the improved U-net full convolution neural network structure built in the step comprises 3 down-sampling layers and 3 up-sampling layers, wherein the characteristic layers output by the 3 down-sampling layers are respectively spliced and fused with the characteristic layers output by the 3 up-sampling layers, namely jump connection is carried out;
further, in 3 down-sampling layers and 3 up-sampling layers of the structure, each down-sampling layer contains two residual convolution blocks and one pooling layer. Each residual volume block comprises a pre-activation layer, a BN (batch normalization) layer and two convolution operations, wherein the two convolution operations are added with the BN (batch normalization) layer and the activation layer, in addition, the convolution kernel size in the convolution operation layer in each residual volume block is 3 multiplied by 3, the convolution kernel size in the pooling layer is 2 multiplied by 2, and the number of the convolution kernels in the convolution layers in 3 downsampling layers is respectively 16, 32 and 64;
the structure in the up-sampling layer is similar to the structure of the corresponding down-sampling layer, the difference is that the pooling layer in the down-sampling layer is replaced by a transposed convolution layer, the convolution kernel size is also 2 x 2, and the number of the convolution kernels in the convolution layers in 3 up-sampling layers is 64, 32 and 16 respectively;
and a connecting layer is also arranged between the last down-sampling layer and the first up-sampling layer and comprises two residual convolution blocks, wherein the number of convolution kernels in each convolution block is 256.
Further, an activation function Mish function is used as an activation function in each of the down-sampling layer and the up-sampling layer, and the formula is as follows:
further, the last convolution operation is performed after the last upsampling layer, the size of a convolution kernel is 1 × 1, the number of the convolution kernels is 1, the convolution operation outputs a feature map with the size of 1024 × 1024 × 1, the feature map obtains a coal rock micro-component class label probability distribution map P through a Sigmoid activation function, and the Sigmoid activation function calculation formula is as follows:
of course, the protection scope of the present invention is not limited thereto, and only one possible technical solution is illustrated here, and those skilled in the art can reasonably design and improve the network model of the Unet according to actual needs;
(3) training and parameter learning of the improved Unet network model:
training the built improved Unet network model by using a training data set and performing K-fold cross validation, specifically comprising:
(3.1) initializing the parameters of the improved Unet network model built in the step (2) by adopting an Xavier initialization method;
(3.2) performing offline data expansion on the coal rock microscopic data set according to the weight ratio of 4: 1: 1, dividing the image data into a training set, a verification set and a test set (the specific setting can be carried out according to actual conditions), training a network model by adopting a five-fold cross verification method, inputting image data into a network in batches in the training process for training, and carrying out online data enhancement and standardized processing by using the step (1) before inputting the image data of each batch into the network model;
(3.3) inputting the gray-scale image of the coal rock microscopic image with the label into the improved Unet network model, as described in the step (2), obtaining a coal rock microscopic component category label probability distribution map P after a Sigmoid activation function, calculating the prediction loss by using the category label probability map and a real label, and particularly adopting focal loss as a target function to balance the problem of proportion imbalance between a normal sample and a difficult sample, wherein the calculation formula is as follows:
FL(pt)＝-αt(1-pt)λlog(pt) Wherein
(3.4) model evaluation index: in the forward reasoning process, the Accuracy, Precision and Dice indexes are calculated by using the obtained and real labels after the Sigmoid activation function, and the specific formula is as follows:
and (3.5) optimizing the focal loss target loss function by adopting a random gradient descent algorithm, and updating the network model parameters by adopting a back propagation algorithm.
The coal rock total component extraction method based on the improved U-net network comprises the following specific post-processing operations:
(1) distinguishing the foreground and the background of the coal rock component category probability distribution map P according to a preset probability threshold value T so as to obtain a determined coal rock component area mask c, wherein the calculation formula is as follows:
(2) performing optimization processing by using a fully connected conditional random field (Dense CRF);
the fully connected conditional random field is an image post-processing mode commonly used in the current deep learning image segmentation application, is an improved mode of the conditional random field, can process classification results obtained by deep learning by combining the relations among all pixels in an original image, optimizes rough and uncertain marks in the classification images, corrects finely-broken wrong regions, and obtains finer segmentation boundaries at the same time.
In this step, the deep full convolution neural network output pixel class label probability distribution map is input into a full-connection conditional random field (density CRF), the class label probability of the pixel is optimized according to the intensity and the position feature similarity between the pixels, the optimized pixel label class probability distribution map is output, and the specific calculation includes:
the fully connected conditional random field model adopts a Gibbs (Gibbs) energy function, and the calculation formula is as follows:
where x is a pixel class label, xi、xjThe category labels corresponding to the ith and jth pixels respectively,
is a function of the unary potential,
is a potential function of pairs;
further, a univariate potential function
The definition is as follows:
wherein, P (x)i) Is the class label prediction probability of the ith pixel output by the deep full convolution neural network.
Further, a potential function of pairs
The definition is as follows:
wherein the content of the first and second substances,
the material is a Gaussian core with the appearance,
to smooth the Gaussian kernel, u (x)i,xj) Is a class tag compatibility function, pi、pjAre the corresponding positions of the ith and jth pixels, Ii、IjThe intensities, σ, corresponding to the ith and jth pixels, respectivelyα、σβAnd σγIs the Gaussian kernel parameter, ω1Nucleus omega2Are the relative intensity coefficients of two gaussian kernels.
(3) Splicing mask images;
due to the limitation of GPU video memory, the original coal rock microscopic image is processed into 1024 × 1024 sub-images through overlapping blocks, so that the mask images of the predicted sub-images need to be spliced in the image post-processing.
(4) Removing a threshold value of the area of the fine crushing area;
and (4) performing area threshold on the coal rock full-component mask image which is spliced in the step (3) to further remove the fragments, wherein the area threshold of the fragments is determined as 1/N of the whole image area of the coal rock microscopic image.
(5) Acquiring a coal rock full-component image by using an image mask:
and after a series of image post-processing, obtaining a segmentation mask image of the coal rock microscopic image to be segmented, and performing masking operation on the original image by using the segmentation mask image to obtain a final coal rock full-component image with background resin and other noises removed.
The present invention also provides an electronic device comprising one or more processors, one or more memories, one or more programs, and an image input device;
the image input device is used for inputting images to be processed, the one or more programs are stored in the memory, and when the one or more programs are executed by the processor, the electronic equipment is enabled to execute the coal rock full component extraction method based on the improved U-net network.
Has the advantages that:
(1) the coal rock total component extraction method based on the improved U-net network overcomes the defects that the existing traditional image segmentation algorithm has weak adaptability, cannot deal with complex coal rock microscopic images and needs manual intervention when performing coal rock image segmentation;
(2) the coal rock total component extraction method based on the improved U-net network can realize effective segmentation of images (can segment resin adhesives and chitin groups in gray overlapping areas and can eliminate pasty and semitransparent bonding formed by coal particle falling off under a polished surface);
(3) the coal rock total component extraction method based on the improved U-net network has good adaptability, does not need manual intervention, and has a great application prospect.
Drawings
FIG. 1 is a schematic flow chart of a coal rock total component extraction method based on an improved U-net network in the invention;
FIG. 2 is a schematic diagram of an original image overlap blocking method according to the present invention;
FIG. 3 is a schematic diagram of residual convolution blocks in a network model used in the present invention;
FIG. 4 is a schematic diagram of the structure of the semantic segmentation network (i.e., the improved U-net network) of the present invention;
FIGS. 5-10 are graphs comparing experimental results of the semantic segmentation network (i.e., the improved U-net network), the FCN network, the U-net network, and the RES-Unet network of the present invention, respectively.
Detailed Description
The technical solutions in the specific embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is to be understood that the described embodiments are some, but not all embodiments of the present invention. The following describes specific embodiments of the present invention, but it will be understood by those skilled in the art that these embodiments are merely illustrative and various changes or modifications may be made therein without departing from the principle and spirit of the present invention, and therefore, the scope of the present invention is defined by the appended claims.
The invention provides a coal rock total component extraction method based on an improved U-net network, which can accurately and quickly remove background resin to achieve the purpose of extracting coal rock total components.
Example 1
A coal rock total component extraction method based on an improved U-net network is shown in figure 1 and comprises the following steps:
s1, shooting the coal rock microscopic image by using a Zeiss Axio-Zoom camera and carrying out image blocking treatment:
the method comprises the steps of shooting a coal rock microscopic image by using a Zeiss Axio-Zoom camera, carrying out overlapping and blocking processing on the coal rock microscopic image to obtain a subgraph with the size of 1024 multiplied by 1024, wherein an overlapping area is 1/10 (namely an overlapping area of 100 pixels) of the subgraph size, and the specific overlapping mode is shown in FIG. 2.
S2, constructing a coal rock image data set, including data set labeling, data set screening and data expansion:
labeling the data set: manually calibrating the coal-rock microscopic image, wherein all coal-rock component areas in the microscopic image are foreground, and non-coal-rock component areas are background (including non-coal-rock component areas such as high-brightness areas caused by mushy, semitransparent cohesiveness and overexposure);
and (3) screening a data set: screening a typical sample to prepare a data set, wherein the data set comprises a normal sample (coal rock components and background resin are clearly distinguished) and a difficult sample (a sample with a resin adhesive and a chitin component overlapped or a pasty and semitransparent bonding phenomenon formed by coal particles falling off from a polished surface);
data expansion: and simultaneously carrying out zooming, translation, mirror image and contrast enhancement on the partitioned coal rock microscopic subgraph and the marked segmentation standard image to obtain a group of transformed coal rock microscopic subgraph and segmentation standard image, thereby expanding the data set of the coal rock microscopic image.
In the step, firstly, the partitioned coal rock microscopic image subgraph is labeled, and a labelme image labeling tool is used for labeling two important categories of the coal rock microscopic image, namely a background resin area and a coal rock component area. And (3) generating file data storing categories and corresponding coordinate points after the labelme marking tool is used for marking the coal-rock microscopic subgraph. And reading the file data, and detecting and labeling the image by using a related function in an OpenCV computer vision library, wherein the background resin area is labeled with black, and the coal rock component area is labeled with white, so that the artificially segmented coal rock full-component labeled image is generated.
And performing data expansion by using the acquired coal rock microscopic image and the manually segmented coal rock full-component labeled image, wherein the specific expansion mode is to perform random scaling, translation, overturning, rotation and contrast enhancement on the conventional image. The scaling of the coal rock microscopic image is within a proper range to prevent excessive scaling, which leads to image distortion and loss of an effective area, the random value range of the scaling factor is [0.8,1.2], the translation size of the image is limited, and the loss of the effective area is prevented. The flipping operation of the image is horizontal and vertical flipping. The rotation operation of the image is mainly mirror image rotation along a vertical axis, and the angle can be selected from 90 degrees, 180 degrees and the like. The image contrast enhancement operation mainly widens the gray scale range of the image by various methods such as gamma conversion and the like, thereby achieving the effect of enhancing the contrast of the coal rock microscopic image. The method is used for carrying out data expansion on the coal rock microscopic image, so that a coal rock microscopic image data set with richer samples is established, the image data after the data expansion can increase the diversity of the characteristics of the samples, and the generalization and the stability of the depth network model are improved.
S3, training an improved deep full convolution neural network Res-Unet-S, comprising: data online expansion and standardization processing, network structure building, model training and parameter learning;
s3.1, online data expansion and standardization:
compared with the data expansion in the step S2, the data expansion used in this step is online data enhancement, that is, online data enhancement in the training process, so that there is a difference in the image data input to the network in each round of training process, thereby further improving the generalization and stability of the semantic segmentation model, and the online data enhancement specifically includes: random mirror inversion, random gamma transformation, random brightness enhancement, random contrast enhancement, random rotation, random translation, and the like.
In addition, the step z-score normalizes the data of model training of the final input network, and the specific calculation process is shown as formula (1), wherein the mean value and the standard deviation are calculated by the overall image data of the data set after offline amplification.
S3.2, building a deep full convolution neural network model:
as shown in fig. 4, the improved U-net full convolution neural network structure constructed in this step includes 3 down-sampling layers and 3 up-sampling layers, wherein the feature layers output by the 3 down-sampling layers are respectively channel-spliced and fused with the feature layers output by the 3 up-sampling layers, that is, jump-connected;
in the 3 down-sampling layers and 3 up-sampling layers of the structure, each down-sampling layer comprises two residual convolution blocks and one pooling layer. Each residual convolution block contains a pre-activation layer, a bn (batch normalization) layer and two convolution operations, both of which are followed by an addition of the bn (batch normalization) layer and an activation layer, and the structure of the residual convolution block is shown in fig. 3. Further, the convolution kernel size in the convolution operation layer in each residual convolution block is 3 × 3, the convolution kernel size in the pooling layer is 2 × 2, and the number of convolution kernels in the convolution layers in the 3 downsampling layers is 16, 32, 64, respectively;
the structure in the up-sampling layer is similar to the structure of the corresponding down-sampling layer, the difference is that the pooling layer in the down-sampling layer is replaced by a transposed convolution layer, the convolution kernel size is also 2 x 2, and the number of the convolution kernels in the convolution layers in 3 up-sampling layers is 64, 32 and 16 respectively;
and a connecting layer is also arranged between the last down-sampling layer and the first up-sampling layer and comprises two residual convolution blocks, wherein the number of convolution kernels in each convolution block is 256.
Further, an activation function Mish function is used as the activation function in each of the down-sampling layer and the up-sampling layer, and the formula is shown in formula (2):
and performing the last convolution operation after the last upsampling layer, wherein the size of a convolution kernel is 1 multiplied by 1, the number of the convolution kernels is 1, the convolution operation outputs a characteristic diagram with the size of 1024 multiplied by 1, the characteristic diagram obtains a coal rock micro-component class label probability distribution diagram P after passing through a Sigmoid activation function, and the Sigmoid activation function calculation formula is shown as a formula (3):
s3.3, model training and parameter learning:
training the built full convolution neural network by using a coal rock microscopic image data set and performing K-fold cross validation, and specifically comprises the following steps:
(1) initializing the parameters of the segmentation network model built in the step S3.2 by adopting an Xavier initialization method;
(2) and (3) performing offline data expansion on the coal rock microscopic data set according to the ratio of 4: 1: 1, dividing the image data into a training set, a verification set and a test set, training a network model by adopting a five-fold cross-validation method, inputting image data into a network in batches in the training process for training, and performing online data enhancement and standardization processing by using a step S3.1 before inputting each batch of image data into the network model;
(3) inputting the gray-scale image of the coal rock microscopic image with the label into a semantic segmentation network, obtaining a coal rock microscopic component class label probability distribution graph P after a Sigmoid activation function as described in step S3.2, calculating a prediction loss by using the class label probability graph and a real label, and specifically adopting focal loss as a target function to balance the problem of proportion imbalance between a normal sample and a difficult sample, wherein the calculation formula is shown as formula (4):
FL(pt)＝-αt(1-pt)λlog(pt) Wherein
(4) Model evaluation and comparative experiments: in the forward reasoning process of the network model, Accuracy, Precision and Dice indexes are calculated by using the obtained and real labels after the Sigmoid activation function, and specific formulas are respectively shown as formulas (5), (6) and (7); as shown in fig. 5, in order to verify the effectiveness of the RES-Unet-S (i.e., the improved U-net network) used in the present invention, a comparison experiment was performed using the FCN network, the Unet network, the RES-Unet network, and the RES-Unet-S network (where the RES-Unet network is different from the improved U-net network of the present application in that the activation function thereof is still the Relu activation function, and the number of down-sampling layers thereof is 4, i.e., the number of down-sampling layers is not reduced), and the superiority of the network structure used in the present invention can be seen from the comparison results of 3 evaluation indexes.
(5) Optimizing a focal loss target loss function by adopting a random gradient descent algorithm, updating network model parameters by adopting a back propagation algorithm, specifically adopting an adam optimization algorithm by adopting an optimization algorithm, and setting the initial learning rate to be 0.0001;
s4, performing coal rock full-component prediction on the image to be segmented by adopting the trained model
Inputting a sample to be tested into a trained full convolution network model for forward propagation reasoning to obtain a coal rock component probability distribution map P, and distinguishing a foreground and a background according to a preset probability threshold value T to obtain a determined coal rock component area mask c, wherein the calculation formula is shown as a formula (8):
s5, carrying out image post-processing on the prediction result of the obtained coal rock component area
S5.1 full-connection conditional random field (Dense CRF) optimization:
the fully connected conditional random field is an image post-processing mode commonly used in the current deep learning image segmentation application, is an improved mode of the conditional random field, can process classification results obtained by deep learning by combining the relations among all pixels in an original image, optimizes rough and uncertain marks in the classification images, corrects finely-broken wrong regions, and obtains finer segmentation boundaries at the same time.
In this step, the deep full convolution neural network output pixel class label probability distribution map is input into a full-connection conditional random field (density CRF), the class label probability of the pixel is optimized according to the intensity and the position feature similarity between the pixels, the optimized pixel label class probability distribution map is output, and the specific calculation includes:
the fully connected conditional random field model adopts a Gibbs (Gibbs) energy function, and the calculation formula is shown as formula (9):
where x is a pixel class label, xi、xjThe category labels corresponding to the ith and jth pixels respectively,
is a function of the unary potential,
is a potential function of pairs;
further, a univariate potential function
The definition is shown in formula (10):
wherein, P (x)i) Is the class label prediction probability of the ith pixel output by the deep full convolution neural network.
Further, a potential function of pairs
The definition is shown in formula (11):
wherein the content of the first and second substances,
the material is a Gaussian core with the appearance,
to smooth the Gaussian kernel, u (x)i,xj) Is a class tag compatibility function, pi、pjAre the corresponding positions of the ith and jth pixels, Ii、IjThe intensities, σ, corresponding to the ith and jth pixels, respectivelyα、σβAnd σγIs the Gaussian kernel parameter, ω1Nucleus omega2Are the relative intensity coefficients of two gaussian kernels.
S5.2 mask image stitching
Due to the limitation of GPU video memory, the original coal rock microscopic image is processed into 1024 × 1024 sub-images through overlapping blocks, so that the mask images of the predicted sub-images need to be spliced in the image post-processing.
S5.3 Fine crushing region area threshold removal
And (4) performing area threshold on the coal rock full-component mask image spliced in the step (S5.2) to further remove the debris, wherein the area threshold of the debris is determined as 1/160000 of the full image area of the coal rock microscopic image.
S5.4 image mask for obtaining coal rock full-component image
And after the series of images are subjected to post-processing, obtaining a segmentation mask image of the coal rock microscopic image to be segmented, and masking the original image by using the segmentation mask image to obtain a final coal rock full-component image with background resin and other noises removed.
The comparison graphs of the experimental results of the comparison experiments of the FCN network, the Unet network, the RES-Unet network and the RES-Unet-S network are shown in fig. 5-10, and it can be known that the improved U-net network model provided by the invention has certain advantages in accuracy, precision and dice indexes compared with other network structures (namely, the FCN network, the Unet network and the RES-Unet network). Although the precision advantage of the method provided by the invention is not obvious enough compared with Unet and RES-Unet, the RES-Unet-S has obvious advantages in terms of model parameters.
The statistics of model parameters of each network are shown in the table below, and it can be seen from the table that the number of model parameters of the RES-uet-S after model pruning is significantly reduced, but comprehensive analysis can see that it still maintains a certain advantage in precision, and the substantial reduction of the number of model parameters further reduces the requirements of the RES-uet-S network on hardware devices, which is beneficial to reducing the hardware cost in practical production application.
Model parameter statistical table
Through verification, the coal rock full-component extraction method based on the improved U-net network overcomes the defects that the existing traditional image segmentation algorithm has weak adaptability, cannot deal with complex coal rock microscopic images and needs manual intervention when performing coal rock image segmentation; the method can realize effective segmentation of the image (can segment the resin adhesive and the shell group in the gray overlapping area and can eliminate paste-like and semitransparent bonding formed by falling of coal particles under a polished surface); the method has good adaptability, does not need manual intervention, and has great application prospect.
Example 2
An electronic device comprising one or more processors, one or more memories, one or more programs, and an image input device;
the image input device is used for inputting images to be processed, one or more programs are stored in the memory, and when the one or more programs are executed by the processor, the electronic equipment is enabled to execute the coal rock full component extraction method based on the improved U-net network according to the embodiment 1.
Although specific embodiments of the present invention have been described above, it will be appreciated by those skilled in the art that these embodiments are merely illustrative and various changes or modifications may be made without departing from the principles and spirit of the invention.
Claims (8)
1. The coal rock full-component extraction method based on the improved U-net network is characterized in that an image to be processed is input into an improved Unet network model after being acquired, the improved Unet network model outputs a coal rock component category probability distribution map, the coal rock component category probability distribution map is subjected to post-processing to obtain a coal rock full-component image, and the post-processing comprises optimization processing by using a full-connection condition random field;
compared with the Unet network model, the improved Unet network model has the improvement that the Mish activation function is adopted to replace the original Relu activation function;
the training process of the improved Unet network model is a process of continuously adjusting model parameters by taking images of a training data set as input and taking coal rock component category labels corresponding to the input as theoretical output, wherein the termination condition of training is that the verification precision of the improved Unet network model is continuously stopped lifting by N rounds when a verification sample model is used for verification, all the images in the training data set are manually calibrated, and the training data set comprises coal rock component category label distribution maps corresponding to all the images.
2. The coal-rock total component extraction method based on the improved U-net network as claimed in claim 1, wherein the improved Unet network model is improved compared with the Unet network model in that a residual volume block in Resnet is adopted to replace an original volume block.
3. The coal-rock full-component extraction method based on the improved U-net network as claimed in claim 1, wherein the training data set is obtained through the following process:
(1) acquiring and preprocessing a coal rock microscopic image;
(2) acquiring a training data set:
(2.1) marking the coal rock microscopic image obtained in the step (1) to finish data set marking;
(2.2) screening a typical sample preparation data set;
and (2.3) performing data expansion on the data set obtained in the step (2.2).
4. The coal rock total component extraction method based on the improved U-net network as claimed in claim 3, wherein the step (1) is as follows:
(1.1) shooting a coal slide by using a camera to obtain an original coal rock microscopic image;
(1.2) carrying out overlapping blocking processing on the original coal rock microscopic image to obtain a subgraph with the size of 1024 multiplied by 1024, wherein the overlapping area is 1/n of the size of the subgraph.
5. The coal-rock full-component extraction method based on the improved U-net network as claimed in claim 3, wherein the specific operation of the labeling is as follows:
manually calibrating the coal rock microscopic image obtained in the step (1), wherein all coal rock component areas in the microscopic image are foreground, and non-coal rock component areas are background;
the specific operation of the data expansion is as follows:
and (3) carrying out random zooming, translation, overturning, rotation and contrast enhancement operation on the coal rock microscopic image marked in the data set obtained in the step (2.2) so as to expand the data set.
6. The coal-rock total component extraction method based on the improved U-net network as claimed in claim 1, wherein the training process of the improved Unet network model is as follows:
(1) data online enhancement and standardization processing;
(2) building an improved Unet network model;
(3) training and parameter learning of the Unet network model is improved.
7. The coal-rock total component extraction method based on the improved U-net network as claimed in claim 1, wherein the post-processing specifically comprises:
(1) distinguishing the foreground and the background of the coal rock component category probability distribution map according to a preset probability threshold value so as to obtain a determined coal rock component area mask;
(2) performing optimization processing by using a full-connection conditional random field;
(3) splicing mask images;
(4) removing a threshold value of the area of the fine crushing area;
(5) and acquiring a coal rock full-component image by using the image mask.
8. An electronic device comprising one or more processors, one or more memories, one or more programs, and an image input device;
the image input device is used for inputting images to be processed, the one or more programs are stored in the memory, and when the one or more programs are executed by the processor, the electronic equipment is enabled to execute the coal rock full component extraction method based on the improved U-net network according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110381605.XA CN113177965A (en) | 2021-04-09 | 2021-04-09 | Coal rock full-component extraction method based on improved U-net network and application thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110381605.XA CN113177965A (en) | 2021-04-09 | 2021-04-09 | Coal rock full-component extraction method based on improved U-net network and application thereof |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113177965A true CN113177965A (en) | 2021-07-27 |
Family
ID=76924725
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110381605.XA Pending CN113177965A (en) | 2021-04-09 | 2021-04-09 | Coal rock full-component extraction method based on improved U-net network and application thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113177965A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113723295A (en) * | 2021-08-31 | 2021-11-30 | 浙江大学 | Face counterfeiting detection method based on image domain frequency domain double-flow network |
| CN116844658A (en) * | 2023-07-13 | 2023-10-03 | 中国矿业大学 | Method and system for rapidly measuring water content of coal based on convolutional neural network |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108062756A (en) * | 2018-01-29 | 2018-05-22 | 重庆理工大学 | Image, semantic dividing method based on the full convolutional network of depth and condition random field |
| CN110807754A (en) * | 2018-08-01 | 2020-02-18 | 华中科技大学 | Fungus microscopic image segmentation detection method and system based on deep semantic segmentation |
| CN111091558A (en) * | 2019-12-12 | 2020-05-01 | 哈尔滨市科佳通用机电股份有限公司 | Railway wagon swing bolster spring jumping fault image identification method |
| CN111160064A (en) * | 2018-11-06 | 2020-05-15 | 煤炭科学技术研究院有限公司 | Coal rock component identification method |
| AU2020103901A4 (en) * | 2020-12-04 | 2021-02-11 | Chongqing Normal University | Image Semantic Segmentation Method Based on Deep Full Convolutional Network and Conditional Random Field |
-
2021
- 2021-04-09 CN CN202110381605.XA patent/CN113177965A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108062756A (en) * | 2018-01-29 | 2018-05-22 | 重庆理工大学 | Image, semantic dividing method based on the full convolutional network of depth and condition random field |
| CN110807754A (en) * | 2018-08-01 | 2020-02-18 | 华中科技大学 | Fungus microscopic image segmentation detection method and system based on deep semantic segmentation |
| CN111160064A (en) * | 2018-11-06 | 2020-05-15 | 煤炭科学技术研究院有限公司 | Coal rock component identification method |
| CN111091558A (en) * | 2019-12-12 | 2020-05-01 | 哈尔滨市科佳通用机电股份有限公司 | Railway wagon swing bolster spring jumping fault image identification method |
| AU2020103901A4 (en) * | 2020-12-04 | 2021-02-11 | Chongqing Normal University | Image Semantic Segmentation Method Based on Deep Full Convolutional Network and Conditional Random Field |
Non-Patent Citations (3)
| Title |
|---|
| SAMAN GHAFFARIAN ET AL: "Post-Disaster Building Database Updating Using Automated Deep Learning: An Integration of Pre-Disaster OpenStreetMap and Multi-Temporal Satellite Data", 《MDPI》 * |
| 司垒 等: "基于改进 U-net 网络模型的综采工作面煤岩识别方法", 《煤炭学报》 * |
| 王洪栋: "基于机器学习的煤岩显微图像分析研究", 《中国优秀博硕士学位论文全文数据库(博士)工程科技Ⅰ辑》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113723295A (en) * | 2021-08-31 | 2021-11-30 | 浙江大学 | Face counterfeiting detection method based on image domain frequency domain double-flow network |
| CN113723295B (en) * | 2021-08-31 | 2023-11-07 | 浙江大学 | Face counterfeiting detection method based on image domain frequency domain double-flow network |
| CN116844658A (en) * | 2023-07-13 | 2023-10-03 | 中国矿业大学 | Method and system for rapidly measuring water content of coal based on convolutional neural network |
| CN116844658B (en) * | 2023-07-13 | 2024-01-23 | 中国矿业大学 | Method and system for rapidly measuring water content of coal based on convolutional neural network |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Xiong et al. | Panicle-SEG: a robust image segmentation method for rice panicles in the field based on deep learning and superpixel optimization | |
| BenTaieb et al. | Adversarial stain transfer for histopathology image analysis | |
| Brancati et al. | A deep learning approach for breast invasive ductal carcinoma detection and lymphoma multi-classification in histological images | |
| CN108364288B (en) | Segmentation method and device for breast cancer pathological image | |
| Yan et al. | Unsupervised image saliency detection with Gestalt-laws guided optimization and visual attention based refinement | |
| Tong et al. | Salient object detection via bootstrap learning | |
| Khan et al. | An efficient contour based fine-grained algorithm for multi category object detection | |
| CN111080628A (en) | Image tampering detection method and device, computer equipment and storage medium | |
| CN110163884B (en) | Single image segmentation method based on full-connection deep learning neural network | |
| CN110633708A (en) | Deep network significance detection method based on global model and local optimization | |
| BenTaieb et al. | Multi-loss convolutional networks for gland analysis in microscopy | |
| CN111539957B (en) | Image sample generation method, system and detection method for target detection | |
| CN111914599B (en) | Fine-grained bird recognition method based on semantic information multi-layer feature fusion | |
| CN113177965A (en) | Coal rock full-component extraction method based on improved U-net network and application thereof | |
| CN111126127A (en) | High-resolution remote sensing image classification method guided by multi-level spatial context characteristics | |
| CN111768415A (en) | Image instance segmentation method without quantization pooling | |
| Gehlot et al. | Ednfc-net: Convolutional neural network with nested feature concatenation for nuclei-instance segmentation | |
| Lv et al. | Nuclei R-CNN: improve mask R-CNN for nuclei segmentation | |
| CN112149689B (en) | Unsupervised domain adaptation method and system based on target domain self-supervised learning | |
| Vahadane et al. | Dual encoder attention u-net for nuclei segmentation | |
| CN116721414A (en) | Medical image cell segmentation and tracking method | |
| Qu et al. | UMLE: unsupervised multi-discriminator network for low light enhancement | |
| CN114548213A (en) | Model training method, image recognition method, terminal device, and computer medium | |
| CN117095180B (en) | Embryo development stage prediction and quality assessment method based on stage identification | |
| CN115423802A (en) | Automatic classification and segmentation method for squamous epithelial tumor cell picture based on deep learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |