CN112381738B - Perspective image self-adaptive correction algorithm based on correlation operation - Google Patents

Perspective image self-adaptive correction algorithm based on correlation operation Download PDF

Info

Publication number
CN112381738B
CN112381738B CN202011324490.2A CN202011324490A CN112381738B CN 112381738 B CN112381738 B CN 112381738B CN 202011324490 A CN202011324490 A CN 202011324490A CN 112381738 B CN112381738 B CN 112381738B
Authority
CN
China
Prior art keywords
image
cross
perspective
noise
correlation
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202011324490.2A
Other languages
Chinese (zh)
Other versions
CN112381738A (en
Inventor
何洋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Qycloud Technology Co ltd
Original Assignee
Shenzhen Qycloud Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Qycloud Technology Co ltd filed Critical Shenzhen Qycloud Technology Co ltd
Priority to CN202011324490.2A priority Critical patent/CN112381738B/en
Priority to PCT/CN2020/138158 priority patent/WO2022105000A1/en
Publication of CN112381738A publication Critical patent/CN112381738A/en
Application granted granted Critical
Publication of CN112381738B publication Critical patent/CN112381738B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/80Geometric correction
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/70Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • G06V10/751Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10004Still image; Photographic image

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses a perspective image self-adaptive correction algorithm based on correlation operation, which relates to the technical fields of computer vision, image processing and signal processing. Compared with the traditional algorithm, the invention has the advantages of higher precision, lower operand and full automation without manual participation, fills up the technical blank in the related fields at home and abroad, and can be applied to computer vision detection scenes in various industries on a large scale.

Description

Perspective image self-adaptive correction algorithm based on correlation operation
Technical Field
The invention relates to the technical field of computer image perspective transformation, in particular to a perspective image self-adaptive correction algorithm based on correlation operation.
Background
With the large-scale popularization of smart phones in recent years, the development of computer vision technology is greatly driven. For example, we can take a picture of the commodity with a mobile phone and then search the matched commodity on the e-commerce platform through the image; for another example, we can scan the text on the paper and then apply OCR (Optical Character Recognition ) techniques to convert the text image into computer recognizable data information.
With the continuous development of deep learning technology, the technology of identifying the external contour, shape, morphology and the like of an object is mature, but at the same time, the technology of extracting detailed information of an image is relatively lagged behind. One of the important factors is that the pictures taken by the mobile phone have a perspective effect. The original form of the image details after perspective is often changed, so that the recognition is difficult. For example, many OCR software has a high recognition rate on the scanned item, but if the scanned item is replaced with a picture taken by a cell phone, the recognition rate is greatly reduced.
Perspective transformation (Perspective Transformation) of images is a common scene in computer vision, which primarily refers to a means of expressing objects in a three-dimensional world onto two-dimensional images. For example, a rectangular paper with written content is placed on a desktop, then the rectangular paper is photographed by a mobile phone from the side, and the originally rectangular paper in the photographed photo is perspective to become a trapezoid, which is called perspective quadrilateral. At this time, if we need to read the content on the paper from the photo by the computer, we have to change the trapezoid in the photo [ wring-out ] into a rectangle, and then the content can be changed into the readable information of the computer. This process is called perspective correction.
The following problems exist in the prior art:
1. the typical practice of perspective correction is that the user is required to keep the vertexes of the perspective quadrangle when photographing, so that 4 vertexes can be positioned through a corner or contour algorithm. However, the limitation of the method is very large, so that the method is not only easy to be influenced by the outside, but also the experience of a user in photographing is poor. Therefore, we need an adaptive perspective correction algorithm, but there is little research on adaptive perspective correction in the current global scope, and most of the published research stays at this level [ affine correction ];
2. in short, affine correction is a process of normalizing a graph subjected to affine transformation, which can be understood as a transformation such as scaling, translation, stretching, etc. of an image only at a two-dimensional level, and does not involve three-dimensional transformation in perspective transformation. For example, when an image is scanned, the paper is clamped by a scanner, which is equal to fixing the paper on a two-dimensional plane, at the moment, affine correction can be applied to angle deviation correction of the scanned image, a periodic noise is embedded in the image in advance, after affine transformation occurs, a plurality of periodic peak point areas are obtained through calculating an autocorrelation matrix, and the arrangement of the peak point areas directly corresponds to the structural trend of affine transformation, so that affine correction can be carried out on the image according to the peak point areas, but the affine correction algorithm cannot meet the perspective correction scene;
3. affine transformation is two-dimensional, perspective transformation is three-dimensional, if the autocorrelation matrix of a perspective image is directly calculated, the obtained peak point areas not only have serious (attenuation), but also have offset positions, so that the distortion of a perspective structure trend template is caused, and therefore, the autocorrelation matrix of the perspective image cannot be calculated as the basis of perspective correction.
Disclosure of Invention
The invention aims to provide a perspective image self-adaptive correction algorithm based on correlation operation, fills the technical blank in the field of computer vision at home and abroad, and realizes accurate self-adaptive perspective correction by fusion application of self-correlation sampling synchronization and template matching technology, unlike a mode of calculating an autocorrelation matrix only adopted by most affine correction researches.
In order to achieve the above purpose, the present invention provides the following technical solutions: a perspective image adaptive correction algorithm based on correlation operation, comprising the steps of:
s1, performing template matching by combining an image processing technology and applying cross-correlation operation of signal processing;
s2, calculating an autocorrelation matrix of periodic noise, and sampling and synchronizing a noise block;
s3, performing cross-correlation calculation on peak points in the autocorrelation matrix of the periodic noise;
s4, performing final template matching perspective correction.
The technical scheme of the invention is further improved as follows: the step S1 further includes:
a. the core computation process of the cross-correlation operation is similar to convolution, assuming that there are two images s (x, y) and f (x, y), and that image s (x, y) is a sub-region of image f (x, y);
b. s (x, y) can be understood as the convolution kernel, and f (x, y) is convolved to calculate [ normalized cross-correlation coefficient ], normalized Cross Correlation Coefficient.
The technical scheme of the invention is further improved as follows: the calculating process of the step a in the step S1 according to the template matching technology is as follows:
a-1, an image s (x, y) is a noise block, and an image f (x, y) is 5×5 identical images s (x, y) are overlapped to form periodic noise;
a-2, knowing that the periodic noise is composed of 25 noise blocks according to the step a-1, namely, template matching can calculate 25 maximum normalized cross correlation coefficients on the periodic noise, and the normalized cross correlation coefficients are called as peak point areas of the periodic noise;
a-3, putting together the peak areas, a structural trend template of the image is formed, and once the image is seen through, the structural trend template is changed correspondingly. In practice, the number of noise blocks matching the image size may be embedded to form a complete structural trend template.
The technical scheme of the invention is further improved as follows: the step S2 of sampling and synchronizing the noise block specifically comprises the following steps:
a. stripping periodic noise from the perspective image;
b. calculating an autocorrelation matrix of the periodic noise;
c. introducing mathematical morphology operation, removing a non-conforming peak point area, and connecting a correct peak point area;
d. contour positioning is carried out on a peak point area in the autocorrelation matrix;
e. performing cross-ratio checking on the contour area, and removing the contour area which does not accord with the cross-ratio;
f. acquiring a measurement parameter of an autocorrelation matrix;
g. and according to the measurement parameters, sampling and synchronizing the noise blocks.
The technical scheme of the invention is further improved as follows: the step S3 of checking peak point cross ratio in the autocorrelation matrix of the periodic noise comprises the following steps:
a. cross-ratio (Cross-ratio) is one of the underlying projection invariants, which can be calculated from 4 collinear points on the projection line;
b. perspective transformation is a common scene of projective transformation, and if collinear points E, F, G, H on a straight line are identical to collinear points E ', F', G ', H' after perspective transformation, their cross ratios are equal.
The technical scheme of the invention is further improved as follows: the step b in the step S3 further comprises the following steps:
the cross-ratio of b-1, H can be calculated using the following formula:
the cross-values of b-2, points E ', F', G ', H' can be calculated using the following formula:
b-3, based on the principle of constant cross ratio, we can therefore derive the following formula:
b-4, points E, F, G, H can be regarded as peak points of periodic noise before perspective transformation, because periodic noise is artificially embedded and known, we can use the values of CR (E, F, G, H) to verify the points E ', F', G ', H' after perspective, i.e. the above formula is just one method of cross ratio calculation, in practical application we can change any group of line segments in 4 collinear points at will, and the characteristic that the cross ratio is constant is still present, for example, the following formula is also true when the cross ratio of peak points is verified:
the technical scheme of the invention is further improved as follows: and in the step S4, final template matching perspective correction is carried out, and the method comprises the following steps of:
a. stripping periodic noise from the perspective image;
b. sampling and synchronizing the noise block;
c. template matching is carried out on the noise block and the periodic noise;
d. introducing mathematical morphology operation, removing a non-conforming peak point area, and connecting a correct peak point area;
e. contour positioning is carried out on a peak point area of the template matching matrix;
f. performing cross-ratio checking on the contour area, and removing the contour area which does not accord with the cross-ratio;
g. generating a perspective structure trend template;
h. and (3) corresponding the coordinates of the structural trend template to the perspective image, and performing perspective correction on the image by using a vertex method.
Compared with the prior art, the invention has the beneficial effects that:
compared with the traditional algorithm, the invention has the advantages of higher precision, lower operand and full automation without manual participation, fills up the technical blank in the related fields at home and abroad, and can be applied to computer vision detection scenes in various industries on a large scale.
Drawings
FIG. 1 is a schematic diagram of a noise block of the present invention;
FIG. 2 is a periodic noise schematic of the present invention;
FIG. 3 is a schematic diagram of the result of template matching according to the present invention;
FIG. 4 is a schematic diagram showing three-dimensional representation of the results after template matching according to the present invention;
FIG. 5 is a block diagram of a noise block sampling synchronization flow in accordance with the present invention;
FIG. 6 is a schematic diagram of a perspective image of the present invention embedded with periodic noise;
FIG. 7 is a block diagram of a template matching perspective correction flow according to the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1-6, the present invention provides a technical solution: a perspective image adaptive correction algorithm based on correlation operation, comprising the steps of:
s1, performing core calculation according to a common template matching method by using cross-correlation operation according to an image processing technology, wherein the method comprises the following steps of:
the core calculation process of the cross-correlation operation is similar to convolution, and assuming that there are two images s (x, y) and f (x, y), and that the image s (x, y) is a sub-region of the image f (x, y), s (x, y) can be understood as a convolution kernel, and the f (x, y) is convolved to calculate [ normalized cross-correlation coefficient ], namely Normalized Cross Correlation Coefficient;
s2, calculating an autocorrelation matrix of periodic noise, and sampling and synchronizing a noise block, wherein the method specifically comprises the following steps:
a. stripping periodic noise from the perspective image;
b. calculating an autocorrelation matrix of the periodic noise;
c. introducing mathematical morphology operation, removing a non-conforming peak point area, and connecting a correct peak point area;
d. contour positioning is carried out on a peak point area in the autocorrelation matrix;
e. performing cross-ratio checking on the contour area, and removing the contour area which does not accord with the cross-ratio;
f. acquiring a measurement parameter of an autocorrelation matrix;
g. according to the measurement parameters, sampling synchronization is carried out on the noise block;
s3, performing cross-correlation calculation on peak points in the autocorrelation matrix of the periodic noise, wherein the cross-correlation calculation comprises the following steps:
a. cross-ratio (Cross-ratio) is one of the underlying projection invariants, which can be calculated from 4 collinear points on the projection line;
b. perspective transformation is a common scene of projective transformation, if collinear points E, F, G and H on a straight line are identical to collinear points E ', F ', G ' and H ' after perspective transformation, the cross ratio of the collinear points E, F, G and H ' is equal;
s4, performing final template matching perspective correction.
Example 1: referring to fig. 1-4, according to the template matching technique, the calculation process for the step a in the step S1 is as follows:
a-1, an image s (x, y) is a noise block, and an image f (x, y) is 5×5 identical images s (x, y) are overlapped to form periodic noise;
a-2, knowing that the periodic noise is composed of 25 noise blocks according to the step a-1, namely, template matching can calculate 25 maximum normalized cross correlation coefficients on the periodic noise, and the normalized cross correlation coefficients are called as peak point areas of the periodic noise;
a-3, putting together the peak areas, a structural trend template of the image is formed, and once the image is seen through, the structural trend template is changed correspondingly. In practice, the number of noise blocks matching the image size may be embedded to form a complete structural trend template.
Example 2: referring to fig. 1-6, step b in step S3 further comprises the following process:
the cross-ratio of b-1, H can be calculated using the following formula:
the cross-values of b-2, points E ', F', G ', H' can be calculated using the following formula:
b-3, based on the principle of constant cross ratio, we can therefore derive the following formula:
b-4, points E, F, G, H can be regarded as peak points of periodic noise before perspective transformation, because periodic noise is artificially embedded and known, we can use the values of CR (E, F, G, H) to verify the points E ', F', G ', H' after perspective, i.e. the above formula is just one method of cross ratio calculation, in practical application we can change any group of line segments in 4 collinear points at will, and the characteristic that the cross ratio is constant is still present, for example, the following formula is also true when the cross ratio of peak points is verified:
example 3: referring to fig. 1-6, final template matching perspective correction is performed in step S4, including the steps of:
a. stripping periodic noise from the perspective image;
b. sampling and synchronizing the noise block;
c. template matching is carried out on the noise block and the periodic noise;
d. introducing mathematical morphology operation, removing a non-conforming peak point area, and connecting a correct peak point area;
e. contour positioning is carried out on a peak point area of the template matching matrix;
f. performing cross-ratio checking on the contour area, and removing the contour area which does not accord with the cross-ratio;
g. generating a perspective structure trend template;
h. and (3) corresponding the coordinates of the structural trend template to the perspective image, and performing perspective correction on the image by using a vertex method.
The beneficial effects of the invention are as follows:
compared with the traditional algorithm, the invention has the advantages of higher precision, lower operand and full automation without manual participation, fills up the technical blank in the related fields at home and abroad, and can be applied to computer vision detection scenes in various industries on a large scale.
Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.

Claims (4)

1. The perspective image self-adaptive correction algorithm based on the correlation operation is characterized by comprising the following steps of:
s1, performing template matching by combining an image processing technology and applying cross-correlation operation of signal processing;
s2, calculating an autocorrelation matrix of periodic noise, and sampling and synchronizing a noise block;
s3, performing cross-correlation calculation on peak points in the autocorrelation matrix of the periodic noise;
s4, performing final template matching perspective correction;
the step S2 of sampling and synchronizing the noise block specifically comprises the following steps:
a. stripping periodic noise from the perspective image;
b. calculating an autocorrelation matrix of the periodic noise;
c. introducing mathematical morphology operation, removing a non-conforming peak point area, and connecting a correct peak point area;
d. contour positioning is carried out on a peak point area in the autocorrelation matrix;
e. performing cross-ratio checking on the contour area, and removing the contour area which does not accord with the cross-ratio;
f. acquiring a measurement parameter of an autocorrelation matrix;
g. according to the measurement parameters, sampling synchronization is carried out on the noise block;
in the step S3, collinear points E, F, G and H on a straight line are identical to collinear points E ', F', G ', H' after perspective transformation, and the cross ratio after perspective transformation is identical;
the perspective transformation also includes the following calculation process:
the cross ratio of b-1 and H is calculated using the following formula:
the cross ratio values of b-2, points E ', F', G ', H' are calculated using the following formula:
b-3, based on the principle of constant cross ratio, obtaining the following formula:
b-4, points E, F, G and H are peak points of the periodic noise before perspective transformation, and the following formula is obtained when the cross ratio of the peak points is calculated:
2. the correlation-based perspective image adaptive correction algorithm according to claim 1, wherein: the step S1 further includes:
a. the core computation process of the cross-correlation operation is similar to convolution, assuming that there are two images s (x, y) and f (x, y), and that image s (x, y) is a sub-region of image f (x, y);
b. s (x, y) can be understood as the convolution kernel, and the normalized cross-correlation coefficient, i.e., normalized cross-correlation coefficient, is calculated by convolving f (x, y).
3. The correlation-based perspective image adaptive correction algorithm according to claim 2, wherein: the calculation process of the step a in the step S1 is as follows:
a-1, an image s (x, y) is a noise block, and an image f (x, y) is 5×5 identical images s (x, y) are overlapped to form periodic noise;
a-2, knowing that the periodic noise is composed of 25 noise blocks according to the step a-1, namely, template matching can calculate 25 maximum normalized cross correlation coefficients on the periodic noise, and the normalized cross correlation coefficients are called as peak point areas of the periodic noise;
a-3, putting together the peak areas, a structural trend template of the image is formed, and once the image is seen through, the structural trend template is changed correspondingly.
4. A perspective image adaptive correction algorithm based on correlation operations according to any one of claims 1-3, characterized in that: and in the step S4, final template matching perspective correction is carried out, and the method comprises the following steps of:
a. stripping periodic noise from the perspective image;
b. sampling and synchronizing the noise block;
c. template matching is carried out on the noise block and the periodic noise;
d. introducing mathematical morphology operation, removing a non-conforming peak point area, and connecting a correct peak point area;
e. contour positioning is carried out on a peak point area of the template matching matrix;
f. performing cross-ratio checking on the contour area, and removing the contour area which does not accord with the cross-ratio;
g. generating a perspective structure trend template;
h. and (3) corresponding the coordinates of the structural trend template to the perspective image, and performing perspective correction on the image by using a vertex method.
CN202011324490.2A 2020-11-23 2020-11-23 Perspective image self-adaptive correction algorithm based on correlation operation Active CN112381738B (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202011324490.2A CN112381738B (en) 2020-11-23 2020-11-23 Perspective image self-adaptive correction algorithm based on correlation operation
PCT/CN2020/138158 WO2022105000A1 (en) 2020-11-23 2020-12-22 Correlation operation-based perspective image adaptive correction algorithm

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011324490.2A CN112381738B (en) 2020-11-23 2020-11-23 Perspective image self-adaptive correction algorithm based on correlation operation

Publications (2)

Publication Number Publication Date
CN112381738A CN112381738A (en) 2021-02-19
CN112381738B true CN112381738B (en) 2023-12-05

Family

ID=74588894

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011324490.2A Active CN112381738B (en) 2020-11-23 2020-11-23 Perspective image self-adaptive correction algorithm based on correlation operation

Country Status (2)

Country Link
CN (1) CN112381738B (en)
WO (1) WO2022105000A1 (en)

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105469428A (en) * 2015-11-26 2016-04-06 河海大学 Morphological filtering and SVD (singular value decomposition)-based weak target detection method

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6873732B2 (en) * 2001-07-09 2005-03-29 Xerox Corporation Method and apparatus for resolving perspective distortion in a document image and for calculating line sums in images
CN108564557B (en) * 2018-05-31 2020-08-25 京东方科技集团股份有限公司 Image correction method and device

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105469428A (en) * 2015-11-26 2016-04-06 河海大学 Morphological filtering and SVD (singular value decomposition)-based weak target detection method

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
景象匹配中的几何透视校正和归一化互相关模板匹配;杨敏等;《航空电子技术》;第34卷(第2期);第38-41页 *

Also Published As

Publication number Publication date
CN112381738A (en) 2021-02-19
WO2022105000A1 (en) 2022-05-27

Similar Documents

Publication Publication Date Title
US8774558B2 (en) Rectification of characters and text as transform invariant low-rank textures
Fua A parallel stereo algorithm that produces dense depth maps and preserves image features
CN110674815A (en) Invoice image distortion correction method based on deep learning key point detection
CN112067233B (en) Six-degree-of-freedom motion capture method for wind tunnel model
CN113808180B (en) Heterologous image registration method, system and device
Serna et al. Data fusion of objects using techniques such as laser scanning, structured light and photogrammetry for cultural heritage applications
CN110956661A (en) Method for calculating dynamic pose of visible light and infrared camera based on bidirectional homography matrix
CN106846249A (en) A kind of panoramic video joining method
CN110120013A (en) A kind of cloud method and device
CN111784587B (en) Invoice photo position correction method based on deep learning network
CN103700082B (en) Image split-joint method based on dual quaterion relative orientation
CN112163995A (en) Splicing generation method and device for oversized aerial photographing strip images
CN113538569A (en) Weak texture object pose estimation method and system
CN112734914A (en) Image stereo reconstruction method and device for augmented reality vision
CN115171138A (en) Method, system and equipment for detecting image text of identity card
CN112381738B (en) Perspective image self-adaptive correction algorithm based on correlation operation
Wang et al. A real-time correction and stitching algorithm for underwater fisheye images
CN115456906A (en) Damaged image-based facial image restoration method, device, equipment and medium
CN109961393A (en) Subpixel registration and splicing based on interpolation and iteration optimization algorithms
CN110674817B (en) License plate anti-counterfeiting method and device based on binocular camera
CN104462111A (en) Image retrieval database establishing method
CN106056599A (en) Object depth data-based object recognition algorithm and device
CN113723465B (en) Improved feature extraction method and image stitching method based on same
Yu et al. Plane-based calibration of cameras with zoom variation
Liao et al. Feature Extraction Method Based on Gray Value Star-shaped Projection

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant