CN111598890B - Level set optimization method for underwater image segmentation - Google Patents

Level set optimization method for underwater image segmentation Download PDF

Info

Publication number
CN111598890B
CN111598890B CN202010410102.6A CN202010410102A CN111598890B CN 111598890 B CN111598890 B CN 111598890B CN 202010410102 A CN202010410102 A CN 202010410102A CN 111598890 B CN111598890 B CN 111598890B
Authority
CN
China
Prior art keywords
level set
function
model
term
underwater image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010410102.6A
Other languages
Chinese (zh)
Other versions
CN111598890A (en
Inventor
王慧斌
陈哲
孙杨
沈洁
张振
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hohai University HHU
Original Assignee
Hohai University HHU
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hohai University HHU filed Critical Hohai University HHU
Priority to CN202010410102.6A priority Critical patent/CN111598890B/en
Publication of CN111598890A publication Critical patent/CN111598890A/en
Application granted granted Critical
Publication of CN111598890B publication Critical patent/CN111598890B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses a level set optimization method for underwater image segmentation, which mainly carries out optimization design on an internal energy function of a level set, namely a regular term, and improves the diffusivity of a regular term diffusion function so as to accelerate the convergence speed and stability of a model; the level set function is normalized through the proposed regular term, so that the reinitialization problem of the model is solved while the character of the symbolic function is maintained, and the segmentation precision lost due to reinitialization is reduced. Compared with the traditional level set method, the improved method has the advantages of higher convergence rate, higher segmentation precision and better robustness for underwater image segmentation.

Description

Level set optimization method for underwater image segmentation
Technical Field
The invention belongs to the technology in the field of image processing, and relates to a level set optimization method for underwater image segmentation.
Background
The reinitialization problem of the level set adds great computational cost to the model, the model can oscillate in the process and the segmentation precision is easily lost, and nowadays, many models are proposed for processing the reinitialization problem of the level set. However, due to the influence of an underwater complex optical environment, the difficulty of underwater image segmentation is greater than that of land image segmentation, and how to overcome the interference of noise in an underwater image on a model becomes a big problem. The conventional regular term model theory is simple in hypothesis and difficult to be used for underwater image segmentation, and noise in an underwater complex scene degrades the function of a regular term, so that the convergence speed of a level set model is slowed down and oscillation is generated.
Oher and Sethian proposed the concept of level set implicit in 1988; malladi and Caselles et al apply this concept in image segmentation in turn; chopp, based on caseles, proposes an energy function and incorporates the level set model in the form of a constraint term. However, the level set function needs to be kept as a symbol distance function in the evolution process, so the level set function needs to be periodically reinitialized to maintain the performance of the model. But the reinitialization of the level set takes a lot of time, increasing the computation cost of the level set function. Based on the M-S model, Chan and Vese et al introduced a homogeneous hypothesis theory and proposed a famous global segmentation model C-V model. The objective function of the model is non-convex and the model still needs to be reinitialized to preserve the regularity of the level set function evolution. Li et al introduced a regularization function based on the model proposed by Caselles, which is essentially based on the level set function of the edge, and after the regularization term is fused, the level set function is kept as a symbol distance function in the evolution process, so that the model gets rid of reinitialization and accelerates the convergence rate, and the model is represented by a DRLSE1 model. However, the model is unstable in value under special conditions and is easy to have negative influence on the model. In 2010, Li et al proposed an optimized distance regularization level set model to overcome adverse effects on the basis of previous work, enhancing the segmentation performance of the model, which is denoted by DRLSE 2. Zhang et al later proposed a Gaussian regularization method, which was introduced into a level set model to achieve local segmentation, and the evolution process of the model was equivalent to the Gaussian smoothing process, thereby avoiding the reinitialization process in the level set evolution. It is worth noting that the model can segment the target object quickly, but there is no further investigation on model oscillation. Then Zhang et al further proposed a new diffusion (RD) method for implicit active contours without re-initialization during the evolution process. The method has good performance on images which are divided into weak edges and easy to generate boundary leakage, but the convergence rate of the method is lower than that of a Gaussian regular level set model.
Disclosure of Invention
The invention aims to: the invention aims to provide a level set optimization method for underwater image segmentation, which improves the stability and convergence speed of model evolution by improving the diffusivity of a diffusion function in a regular term, solves the problems of reinitialization and slow convergence of a level set and further optimizes the segmentation effect of an underwater image.
The technical scheme is as follows: in order to achieve the above object, the present invention provides a level set optimization method for underwater image segmentation, comprising the following steps:
(1) regular term optimization design, the mode of the integrand in the regular term with gradient
Figure GDA0003687415560000021
Is a variable, in
Figure GDA0003687415560000022
And
Figure GDA0003687415560000023
in two intervals, with
Figure GDA0003687415560000024
To constrain
Figure GDA0003687415560000025
Approach to 1 and lie in
Figure GDA0003687415560000026
In the interval (c), utilize
Figure GDA0003687415560000027
And
Figure GDA0003687415560000028
multiply and accelerate
Figure GDA0003687415560000029
A speed approaching 1;
(2) construction of level set energy functional E Z ＝E W +E N Wherein the regularization term E N Internal energy terms, data items E, forming energy functional W An external energy term constituting an energy functional;
(3) initial level set function of phi 0 ＝cos(r 1 x)cos(r 2 y) in which r 1 Is the product of the height of the input underwater image and pi, r 2 The product of the width of the input underwater image and pi, wherein (x, y) is the coordinate of a pixel point in the underwater image;
(4) according to
Figure GDA00036874155600000210
Updating a level set function, and improving the diffusivity of the function by using the provided regular term in the evolution process to accelerate the convergence speed of the model, wherein t is time, delta t represents a time step, and n represents iteration times;
(5) and (4) judging whether the level set function phi reaches a convergence condition, and if the level set function phi does not reach the convergence condition, skipping to the step (4) to continue evolving the level set.
The optimization regularization term in the step (1) aims to solve the problem of reinitialization and improve convergence rate, and the function of the optimization regularization term is defined as:
Figure GDA00036874155600000211
where Ω represents the entire image domain.
Of the regular term functions, the integrand
Figure GDA00036874155600000212
The expression of (a) is:
Figure GDA0003687415560000031
solving an evolution equation by using an Euler-Lagrange equation for the optimization regularization term:
Figure GDA0003687415560000032
the diffusivity function in the optimization regularization term is:
Figure GDA0003687415560000033
the energy functional synthesis external term E in the step (2) W And an internal item E N The latter overall function E Z Comprises the following steps:
Figure GDA0003687415560000034
wherein the content of the first and second substances,
Figure GDA0003687415560000035
as data items, ω i Is a conditioning parameter, S is a saliency value extracted from an underwater image, sa i Is the average of the saliency inside and outside the contour,i is 1,2 is used as subscript to distinguish the inner and outer regions of the contour, g represents the edge weight, and H (Φ) is the hervedside function.
The level set function is solved by a gradient descent method to obtain an evolution equation:
Figure GDA0003687415560000036
and (4) updating and iterating according to the level set evolution equation.
Has the advantages that: the method improves the regular term aiming at the problems of reinitialization and slow convergence, improves the diffusivity of the diffusion function of the regular term, accelerates the convergence speed of the model, and reduces the convergence times required by the curve convergence to the target boundary; and the level set function is regularized through the proposed regular term, the reinitialization problem of the model is solved, and the segmentation precision lost due to reinitialization is further reduced. Experiments show that the level set method provided by the invention can accurately realize underwater image segmentation. Compared with the existing level set method, the method has higher segmentation precision and faster convergence rate for the underwater image.
Drawings
FIG. 1 is a flow chart of an underwater image segmentation method in the present invention.
Fig. 2 is a graph of the underwater original image (a) and the segmentation result (b) when there is no regularization term.
Fig. 3 is a diagram of an underwater original image (a) and a division result (b) when a regularization term is present.
FIG. 4 is a graph comparing the diffusivity equation for the regularization term in the present invention with the diffusivity equation for the regularization term in the DRLSE2 model. The dashed line represents the DRLSE2 diffusivity equation for
Figure GDA0003687415560000041
Curve (c) of (d). The solid line represents the diffusivity equation for the model of the present invention with respect to
Figure GDA0003687415560000042
Curve (c) of (d).
FIG. 5 is a zero level set contour result graph of the present invention method and several comparative methods for underwater image segmentation. Where (a) is the underwater original image, (b) is the result of the DRLSE2 method, (c) is the result of the RD method, (d) is the result of the method of the present invention, and (1) - (8) rows represent underwater images of different objects.
FIG. 6 is a graph of the binarization results of the method of the present invention and several comparison methods for underwater image segmentation. Where (a) is the underwater original image, (b) is the artificial truth labeling, (c) is the result of the DRLSE2 method, (d) is the result of the RD method, (e) is the result of the method of the present invention, and (1) - (8) lines represent underwater images of different objects.
Detailed Description
In order to clearly highlight the objects and advantages of the present invention, the present invention will be further described below with reference to the drawings in the examples of the present invention, and the implementation process of the level set optimization method for underwater image segmentation disclosed in the embodiments of the present invention mainly includes the following steps:
(1) an underwater image such as any one of the images in the (a) column in fig. 5 is input. The optimized regular term function designed by the invention and the image characteristics extracted from the underwater image respectively form an internal energy function and an external energy function of the level set energy functional, wherein the external energy function is formed by the region level characteristics and the edge level characteristics. Wherein the region-level features refer to the intensity of the image. Edge level features refer to the gradients of an image.
The expression of the level set model framework is as follows:
E Z ＝E W +E N
in the formula, E W Is a constraint term representing an external energy term. E N Represents an internal energy function, i.e. a regularization term.
Internal energy function E in the level set model N The expression is as follows:
Figure GDA0003687415560000043
wherein
Figure GDA0003687415560000044
The norm representing the gradient, (x, y) are the coordinates of the pixel points in the underwater image, Ω represents the entire image domain,
Figure GDA0003687415560000045
the functional expression is:
Figure GDA0003687415560000051
in that
Figure GDA0003687415560000052
And
Figure GDA0003687415560000053
on the two intervals, the number of the interval,
Figure GDA0003687415560000054
all are provided with
Figure GDA0003687415560000055
This term constrains the gradient
Figure GDA0003687415560000056
Approaching 1. In addition, the models are not used singly
Figure GDA0003687415560000057
The constraint term is as follows
Figure GDA0003687415560000058
On the interval of (1), model utilization
Figure GDA0003687415560000059
This term and
Figure GDA00036874155600000510
multiply so that
Figure GDA00036874155600000511
The approach to 1 is faster.
Internal energy function E in the level set model N By a regularization term
Figure GDA00036874155600000512
And solving to obtain an evolution equation:
Figure GDA00036874155600000513
according to the regular term function, the diffusivity equation can be obtained as follows:
Figure GDA00036874155600000514
to obtain:
Figure GDA00036874155600000515
combining the diffusivity equation with fig. 4, we find the relevant properties of the diffusivity equation as follows:
1) in that
Figure GDA00036874155600000516
The diffusivity function value h is greater than 0, the function is expressed as forward diffusion, and the model makes the gradient
Figure GDA00036874155600000517
Suppressed to 1.
2) In that
Figure GDA00036874155600000518
Is less than 0, then the function is expressed as back diffusion and the model will be
Figure GDA00036874155600000519
Increasing to 1.
3) In that
Figure GDA00036874155600000520
The diffusivity function value h is greater than 0, the function is expressed as forward diffusion, and the model will be
Figure GDA00036874155600000521
Suppressed to 0. As described above, the level set function can be increased or decreased by such adaptation
Figure GDA00036874155600000522
And minimization of the energy function, which in turn drives the model curve to shrink to the desired target boundary.
As shown in FIG. 4, the diffusivity function of the regularization term of the present invention is compared with that of DRLSE2 model by h 2 And h 1 It is shown, to conclude:
1) in that
Figure GDA0003687415560000061
In the interval, the positive and negative of the function values are forward diffusion, and the function value h 2 Greater than h 1 Represents the proposed function pair
Figure GDA0003687415560000062
The restraining force is greater than DRLSE2, which makes
Figure GDA0003687415560000063
The speed of the model is faster, namely the model has a faster convergence speed when being used for segmenting the underwater image, and meanwhile, the model only needs less iteration times.
2) In that
Figure GDA0003687415560000064
In the interval, all belong to backward diffusion, and the function value h 2 Is still greater than h 1 Similarly, the proposed function pair
Figure GDA0003687415560000065
Is greater than DRLSE2, such that
Figure GDA0003687415560000066
Is faster.
3) For the interval
Figure GDA0003687415560000067
Both the proposed function and the DRLSE2 model guarantee the symbol distance function property far away from the zero level set
Figure GDA0003687415560000068
The external energy function E in the level set model W Expressed as:
Figure GDA0003687415560000069
in the model, g represents the edge weight and the level set function phi represents the curve profile. Omega i Is a conditioning parameter, H (phi) denotes the Hervesseld function, S is a saliency value extracted from an underwater image, sa i The average value of the saliency of the inside and the outside of the outline is used, i is used as a subscript to distinguish the inside and the outside of the outline, and (x, y) are pixel point coordinates in the underwater image.
The expression of the edge detector in the level set model is as follows:
Figure GDA00036874155600000610
wherein I is an input image, G σ Is a gaussian kernel with a standard deviation of sigma,
Figure GDA00036874155600000611
for gradient operation, a convolution operation is denoted.
The Heaviside function is:
Figure GDA00036874155600000612
the region features in the level set model are extracted by a significance method:
S(x,y)＝|I(x,y)-I Gass (x,y)|
wherein I is an input image, I Gass Representing the image after processing with a gaussian filter.
In the region feature extraction method I Gass Is defined as follows:
I Gass (x,y)＝κ ζ *I(x,y)
k in the extraction gaussian image equation is a gaussian kernel with a standard deviation ζ.
The region characteristics in the level set model are extracted by a significance method, and the expression of the significance inside and outside the outline is as follows:
Figure GDA0003687415560000071
Figure GDA0003687415560000072
and finally, the energy functional of the level set model is as follows:
Figure GDA0003687415560000073
(2) initializing a level set function, evolving a level set model, and enhancing the segmentation performance of the model on the underwater image by using the optimized regular term.
The initial level set function is:
φ 0 ＝cos(r 1 x)cos(r 2 y)
r in the initial function 1 For inputting the product of the height of the underwater image and pi, r 2 The product of the width of the underwater image and pi is input, and (x, y) is the coordinate of a pixel point in the underwater image.
The model is solved by a gradient descent method:
Figure GDA0003687415560000074
the energy function is subjected to partial derivation to obtain:
Figure GDA0003687415560000075
wherein div represents divergence, and solving the energy functional to obtain a level set evolution equation:
Figure GDA0003687415560000081
the present invention uses a quantitative standard Jaccard Similarity (JS) standard to accurately evaluate the segmentation performance of these methods. JS Standard, theoretically two sets P 1 And P 2 The ratio of the intersection of (a) to their union is given by:
Figure GDA0003687415560000082
the closer the JS value is to 1, the more P is represented 1 The closer to P 2 The higher the segmentation accuracy of the model. P 1 Represents the result of the segmentation, and P 2 Representing the standard value of the manual annotation.
The selected underwater image samples are divided into two types in terms of characteristics, one type is represented by fuzzy image details and less background interference, and the degree of distinction between a target and a background in the images is lower; one category appears to be strong noise, and objects in such images are often mixed with complex background interference, which brings great challenges to the segmentation model.
TABLE 1 quantitative comparison table of level set model segmentation results
Figure GDA0003687415560000083
It can be seen from fig. 2 and fig. 3 that the segmentation result of the present invention is significantly better than the segmentation result of the model without the regularization term, and although the target region is extracted from the result of the model without the regularization term, the model contains a large amount of point-like noise, and the regularization term of the present invention makes the model separate from the reinitialization, so that the accuracy is not lost.
Qualitative results fig. 5 further illustrates that the present invention achieves its theoretical effect, has better convergence, and has better segmentation effect on underwater images. In order to fully illustrate the auxiliary effect of the proposed regularization term on the model, the invention quantifies and compares the results of qualitative experiments. From table 1 it can be seen that the JS value score of the present invention is first, illustrating the superiority of the method of the present invention over the two comparison algorithms. In addition, the calculation time and convergence times of the model evolution are counted. The run time results are shown in table 1, the run times are the CPU time required for the model to complete convergence, and it can be seen that the time spent by the model of the present invention is also ranked first.
In addition, on the premise of ensuring that each algorithm obtains the optimal result, comparing the iteration times of completing the segmentation of the algorithms, the number of the DRLSE2 models is 320, the number of the RD models is 400, and the iteration times of the method is 40. The RD algorithm, although more accurate than the DRLSE2 method in segmentation, iterates slower on this model. The method is obtained by integrating two indexes of the segmentation precision JS and the iteration times, is superior to a comparison method, and can prove the excellent performance of the method.

Claims (4)

1. A level set optimization method for underwater image segmentation is characterized by comprising the following steps: comprises the following steps:
(1) the regular term is optimally designed, and the regular term is defined as
Figure FDA0003687415550000011
Wherein the content of the first and second substances,
Figure FDA0003687415550000012
representing the modulus of the gradient, Ω denotes the entire image domain, the integrand
Figure FDA0003687415550000013
The expression of (a) is:
Figure FDA0003687415550000014
in that
Figure FDA0003687415550000015
And
Figure FDA0003687415550000016
in two intervals, with
Figure FDA0003687415550000017
To constrain
Figure FDA0003687415550000018
Approach to 1 and lie in
Figure FDA0003687415550000019
In the interval of (1), use
Figure FDA00036874155500000110
And
Figure FDA00036874155500000111
multiply and accelerate
Figure FDA00036874155500000112
A speed approaching 1;
(2) construction of level set energy functional E Z ＝E W +E N Wherein the regularization term E N Is an internal energy term of an energy functional, data item E W Is an external energy term of the energy functional;
(3) initial level set function of phi 0 ＝cos(r 1 x)cos(r 2 y) in which r 1 Is the product of the height of the input underwater image and pi, r 2 The product of the width of the input underwater image and pi, wherein (x, y) is the coordinate of a pixel point in the underwater image;
(4) according to the equation
Figure FDA00036874155500000113
Updating a level set function, improving diffusivity by using an optimization regular term in an evolution process, and accelerating the convergence speed of a model, wherein t is time, delta t represents a time step, and n represents iteration times; a function of diffusivity of
Figure FDA00036874155500000114
(5) And (4) judging whether the level set function phi reaches a convergence condition, and if the level set function phi does not reach the convergence condition, skipping to the step (4) to continue evolving the level set.
2. The method for optimizing the level set of underwater image segmentation as claimed in claim 1, wherein: solving an evolution equation by using an Euler-Lagrange equation for the optimization regularization term as follows:
Figure FDA00036874155500000115
wherein div represents the divergence.
3. The method of claim 1, wherein the method comprises: the energy functional synthesis external term E in the step (2) W And an internal item E N The latter overall function E Z Comprises the following steps:
Figure FDA0003687415550000021
wherein the content of the first and second substances,
Figure FDA0003687415550000022
as data items, ω i Is a conditioning parameter, S is a saliency value extracted from an underwater image, sa i Is the average of the saliency inside and outside the contour, i ═ 1,2 is used as a subscript to distinguish between the inside and outside regions of the contour, g represents the edge weight, and H (Φ) is the hervesseld function.
4. The method of claim 3, wherein the level set optimization is performed by: after the level set function is solved by a gradient descent method, a level set evolution equation is obtained as follows:
Figure FDA0003687415550000023
wherein div represents divergence, and updating iteration is performed according to the level set evolution equation in the step (4).
CN202010410102.6A 2020-05-15 2020-05-15 Level set optimization method for underwater image segmentation Active CN111598890B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010410102.6A CN111598890B (en) 2020-05-15 2020-05-15 Level set optimization method for underwater image segmentation

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010410102.6A CN111598890B (en) 2020-05-15 2020-05-15 Level set optimization method for underwater image segmentation

Publications (2)

Publication Number Publication Date
CN111598890A CN111598890A (en) 2020-08-28
CN111598890B true CN111598890B (en) 2022-08-26

Family

ID=72183715

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010410102.6A Active CN111598890B (en) 2020-05-15 2020-05-15 Level set optimization method for underwater image segmentation

Country Status (1)

Country Link
CN (1) CN111598890B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113093198B (en) * 2021-03-10 2023-07-25 河海大学 Acoustic imaging detection method based on multi-scale Markov random field

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7630530B2 (en) * 2005-01-14 2009-12-08 Siemens Medical Solutions Usa, Inc. System and method for fast tensor field segmentation
CN105976378B (en) * 2016-05-10 2019-05-10 西北工业大学 Conspicuousness object detection method based on graph model
CN106296695B (en) * 2016-08-12 2019-05-24 西安理工大学 Adaptive threshold natural target image segmentation extraction algorithm based on conspicuousness
CN110136146B (en) * 2019-05-17 2021-01-08 浙江理工大学 SAR image water area segmentation method based on sinusoidal SPF distribution and level set model
CN111105430B (en) * 2019-11-28 2023-08-08 青岛大学 Variation level set image segmentation method based on Landmark simplex constraint

Also Published As

Publication number Publication date
CN111598890A (en) 2020-08-28

Similar Documents

Publication Publication Date Title
US20220414891A1 (en) Method for automatic segmentation of fuzzy boundary image based on active contour and deep learning
WO2021017361A1 (en) Template matching algorithm based on edge and gradient feature
US9959466B2 (en) Object tracking apparatus and method and camera
Andersson et al. Modified gradient search for level set based image segmentation
CN111681300B (en) Method for obtaining target area composed of outline sketch lines
CN111598890B (en) Level set optimization method for underwater image segmentation
CN110120057A (en) Fuzzy region sexuality contours segmentation model based on the global and local fitting energy of weight
Yang Local smoothness enforced cost volume regularization for fast stereo correspondence
Yu et al. A survey of level set method for image segmentation with intensity inhomogeneity
Huang et al. Variational level set method for image segmentation with simplex constraint of landmarks
CN107424153B (en) Face segmentation method based on deep learning and level set
Gao et al. Fast multiregion image segmentation using statistical active contours
CN105139398A (en) Multi-feature-based gray uneven image fast segmentation method
Badshah et al. On two multigrid algorithms for modeling variational multiphase image segmentation
CN117173201A (en) Second order differential image segmentation method, system, medium and device
CN111105430A (en) Variation level set image segmentation method based on Landmark simplex constraint
Bao et al. Solar panel segmentation under low contrast condition
CN114596320A (en) Image segmentation method and device based on ALSLCV model
Zia et al. Active Contour Model for Image Segmentation
Sanou et al. Semi-automatic tools for nanoscale metrology and annotations for deep learning automation on electron microscopy images
Zhang et al. Divergence of gradient convolution: Deformable segmentation with arbitrary initializations
Wu et al. Superpixel based hierarchical segmentation for color image
CN112700459A (en) Level set infrared image segmentation method based on multi-feature information fusion
CN117593323B (en) Image segmentation method, system, medium and device based on non-local features
CN112102350A (en) Secondary image segmentation method based on Otsu and Tsallis entropy

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant