CN112541092A - Three-dimensional image contour retrieval method and system based on tangential domain and storage medium - Google Patents

Three-dimensional image contour retrieval method and system based on tangential domain and storage medium Download PDF

Info

Publication number
CN112541092A
CN112541092A CN202011563813.3A CN202011563813A CN112541092A CN 112541092 A CN112541092 A CN 112541092A CN 202011563813 A CN202011563813 A CN 202011563813A CN 112541092 A CN112541092 A CN 112541092A
Authority
CN
China
Prior art keywords
contour
dimensional image
sequence
tangential
domain
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202011563813.3A
Other languages
Chinese (zh)
Other versions
CN112541092B (en
Inventor
杜娟
闫甜甜
胡晓明
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
South China University of Technology SCUT
Original Assignee
South China University of Technology SCUT
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by South China University of Technology SCUT filed Critical South China University of Technology SCUT
Priority to CN202011563813.3A priority Critical patent/CN112541092B/en
Publication of CN112541092A publication Critical patent/CN112541092A/en
Application granted granted Critical
Publication of CN112541092B publication Critical patent/CN112541092B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/13—Edge detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G06T2207/10012—Stereo images

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Library & Information Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a tangential domain-based three-dimensional image contour retrieval method, a system and a storage medium, which comprises the steps of clustering K-means to generate a feature view of a projection view of a three-dimensional image and extracting the contour of the feature view. Then, the Gaussian kernel function is used for convolution and is converted into a tangential domain for retrieval. The method can eliminate the influence of the starting point on the contour retrieval, has the invariance of rotation, translation, scale and mirror image, has low feature dimension and high extraction speed, and can obtain higher retrieval rate and better universality compared with other contour retrieval algorithms.

Description

Three-dimensional image contour retrieval method and system based on tangential domain and storage medium
Technical Field
The invention relates to the field of image processing, in particular to a three-dimensional image contour retrieval method and system based on a tangential domain and a storage medium.
Background
The contour of an object is an inherent attribute and does not change along with illumination, color and texture, so a contour-based target identification method is researched in a large quantity, but the contour of the object can be presented differently on an image along with the change of a shooting angle, the extracted contour is deformed due to the movement and the shielding of a non-rigid object, the same type of object also has different properties, the problems bring huge challenges to related researches, and the research scheme needs to make good balance among speed, precision and robustness.
The conventional three-dimensional image-oriented research scheme can directly establish a descriptor from a three-dimensional image and can also indirectly establish the descriptor from the projection of the three-dimensional image, so that the rapidity and the effectiveness of three-dimensional image retrieval are met, the feature dimension can be reduced from the aspect of feature extraction, the feature extraction time is further reduced, the efficiency is improved, a better matching algorithm can be designed from the aspect of feature matching, and the effectiveness of a retrieval result is enhanced.
Disclosure of Invention
In order to overcome the defects and shortcomings of the prior art, the invention provides a three-dimensional image contour retrieval method, a three-dimensional image contour retrieval system and a storage medium based on a tangential domain.
The invention adopts the following technical scheme;
a three-dimensional image contour retrieval method based on a tangential domain comprises the following steps:
generation of S1 projection diagram: inputting a three-dimensional image, zooming and translating the three-dimensional image to the center of a unit sphere, generating viewpoints on the unit sphere, and generating a projection drawing at the position of each viewpoint;
s2 feature view extraction: extracting the sequence contour of each projection drawing, and recording the sequence contour as CiWherein i is the number of pictures and uses a one-dimensional equation CiExpressed as x (t) + jy (t), for CiPerforming Fourier series expansion, selecting the first n Fourier coefficients for description, randomly selecting four clustering centers, clustering by using a K-means algorithm, and taking an image closest to the clustering centers as a characteristic view of the image;
s3 arc length equal resampling: counting the number of pixels of the characteristic view as the total arc length according to
Figure BDA0002860086890000011
Carrying out arc length parameterization, wherein m represents the arc length of the position of the sequence contour point, carrying out cubic B-spline interpolation and carrying out arc length equal interval resampling to obtain a sequenceA column profile;
s4, convolving the sequence contour obtained in the last step with a Gaussian kernel function;
s5 tangential domain feature extraction: t ═ x (s, σ)max)′，y(s，σmax) '), mapping T to a unit circle, dividing the unit circle according to a certain angle interval margin, counting the number n (k) of points in each bin, wherein k is 1,2maxAnd greater than 0.8nmaxThe angles are respectively used as a main direction and an auxiliary direction, and the characteristic vector is converted to the main direction fmain(i) And an auxiliary direction fau(i) Then mirror the features fmir(i) To obtain CiIs noted as fj(j＝1，2，3，4；)，fj＝{fmain(i)，fau(i)，fmir(i) J represents the number of feature contours;
s6 defines Dpq:
Figure BDA0002860086890000021
Figure BDA0002860086890000022
[·]representing and merging into a vector;
s7 defines the threshold and retrieves: will DpqAnd according to the sequence from small to large, taking the maximum value in the first four-dimensional minimum values of a certain class as a threshold value for searching the image.
Further, in S1, a viewpoint is generated on the unit sphere by using the gaussian seidel iteration.
Further, the number of viewpoints is 200.
Further, in S2, four feature views are generated.
Further, in S3, the number of resample points is 400.
Further, due to the translational invariance of the tangential domain and the one-to-one mapping between the closed sequence contour and the tangential domain, C will beiConversion to the tangential domain for expression and feature extraction.
Further, the sequence contour of each projection image is extracted by adopting a morphological watershed algorithm.
A system for realizing the three-dimensional image contour retrieval method comprises the following steps:
the projection graph generation module: inputting a three-dimensional image, zooming and translating the three-dimensional image to the center of a unit sphere, generating viewpoints on the unit sphere, and generating a projection drawing at the position of each viewpoint;
the characteristic view extraction module: extracting the sequence contour of each projection drawing, and obtaining a characteristic view of the projection drawing after processing;
arc length equal interval sampling module: counting the number of pixels of the characteristic view as the total arc length, and performing resampling on the arc length at equal intervals to obtain a sequence profile;
a convolution module: convolving the obtained characteristic view sequence outline with a Gaussian kernel function;
a tangential domain feature extraction module: obtaining a feature vector of the projection drawing;
the retrieval module: definition DpqAnd according to the sequence from small to large, taking the maximum value in the first four-dimensional minimum values of a certain class as a threshold value for searching the image.
A storage medium having stored thereon a computer program that executes the three-dimensional image contour detection method.
The invention has the beneficial effects that:
(1) the invention analyzes the projection of the three-dimensional image under different visual angles by the three-dimensional image contour retrieval method based on the tangential domain, improves the retrieval efficiency of the three-dimensional image, and has simple process and low calculation complexity.
(2) The method has stronger robustness, and can achieve good retrieval results for the property change of the objects in the class.
(3) The method has strong applicability, not only can be used for searching three-dimensional images, but also can be popularized to searching two-dimensional images, and the classifier can be designed according to the extracted characteristics for identifying objects.
Drawings
FIG. 1 is a flow chart of the operation of the present invention.
Detailed Description
The present invention will be described in further detail with reference to examples and drawings, but the present invention is not limited to these examples.
Example 1
As shown in fig. 1, a tangential domain-based three-dimensional image contour retrieval method of the present invention includes the following steps:
(1) projection view generation: inputting a three-dimensional image I, zooming and translating the three-dimensional image I to the center of a unit sphere, generating 200 viewpoints on the unit sphere by Gauss Seidel iteration, and generating a projection diagram at the position of each viewpoint.
The viewpoints generated by the method are more uniform than longitude and latitude coordinates, and 200 viewpoints not only can contain rich detail information, but also do not occupy more calculation time.
(2) Extracting a characteristic view: the sequence contour of each projection graph is extracted by the morphology watershed algorithm and is marked as CiI 1,2 …,200, using one-dimensional equation CiExpressed as x (t) + jy (t), for CiAnd performing Fourier series expansion, selecting the first 20 Fourier coefficients for description, randomly selecting 4 clustering centers, clustering by using a K-means algorithm, and taking the image closest to the clustering centers as a characteristic diagram of the image.
The four characteristic views generated in the step have no difference between the optimal view and the worst view, the subsequent matching process is to synthesize the matching results of the four views, and the characteristic views are generated according to K-means clustering, so that the redundancy of projection views is reduced, the most representative projection view is extracted, and the retrieval rate and the retrieval speed are improved.
In this embodiment, four selected empirical values are selected for The clustering center, and other numerical values may be selected according to specific situations, and The Princeton Shape Benchmark dataset is generally used.
(3) Arc length equal resampling: counting the number of pixels as the total arc length according to
Figure BDA0002860086890000041
Carrying out arc length parameterization, wherein m represents the arc length of the position of the sequence contour point, carrying out cubic B-spline interpolation, carrying out arc length and the likeAnd re-sampling at intervals, and selecting 400 re-sampling points to obtain the sequence profile.
The step can ensure the second-order conductibility of the contour and the scale invariance of subsequent feature extraction, the number of the resampling points is 400, on one hand, the number can ensure that the error of the contour after sampling is small, and on the other hand, the algorithm has higher retrieval speed.
(4) Gaussian kernel function convolution: defining a Gaussian kernel function of
Figure BDA0002860086890000042
The window width can be changed by adjusting the standard deviation σ, and the sequence contour obtained in the previous step is convolved with the kernel function, x (s, σ) ═ x(s) · Gauss (σ), y (s, σ) ═ y(s) · Gauss (σ).
(5) Extracting features of a tangential domain: t ═ x (s, σ)max)′，y(s，σmax) '), mapping T onto a unit circle, dividing the unit circle by a certain angular interval margin (e.g., margin pi/18), counting the number of points n (k) in each bin, k being 1,2maxAnd greater than 0.8nmaxThe angles are respectively used as a main direction and an auxiliary direction, and the characteristic vector is converted to the main direction fmain(i) And an auxiliary direction fau(i) Note that the secondary direction is not unique, and then the features are mirrored fmir(i) Then C is obtainediIs noted as fj(j＝1，2，3，4；)，fj＝{fmain(i)，fau(i)，fmir(i) J denotes the number of feature contours.
Tangential domain transformation, C, due to translational invariance of the tangential domain and one-to-one mapping between the closed sequence contours and the tangential domainiConversion to the tangential domain for expression and feature extraction.
Because it is greater than 0.8nmaxMay be more than one, and thus the secondary directions are not unique, it is necessary to calculate the feature vector in each secondary direction, with fauTo indicate.
(6) Definition Dpq：
Figure BDA0002860086890000043
Figure BDA0002860086890000044
[·]Are represented and are combined into a vector.
DpqA distance matrix representing the three-dimensional images p and q.
(7) Defining a threshold and retrieving: will DpqAnd according to the sequence from small to large, taking the maximum value in the first four-dimensional minimum values of a certain class as a threshold value for searching the image.
Taking The Princeton Shape Benchmark dataset as an example, The data set has 90 classes in total, and The number of images in each class is not The same.
Because each three-dimensional image has four characteristic views, the existing matching algorithm can not be directly matched, and D is definedpqTo perform a search for a three-dimensional image.
Search is conducted if DpqIf the current graph is smaller than the threshold value, the current graph is output, otherwise, the next graph is input for retrieval again.
One application example of this embodiment is as follows:
a three-dimensional image is input, the data format of which is a point and a surface, the surface is composed of three points, and therefore the three-dimensional image is a connection relationship between a three-dimensional coordinate point and a point.
200 projection views are obtained, and 4 characteristic view outlines after clustering are obtained, wherein the number of the outline points is 2689 pixels in the example.
After the arc length equal resampling, the number of each feature profile point is 400, the sequence profile is convolved with the gaussian kernel function, and the standard deviation σ is set to 15.
Extracting tangential domain features, taking an input picture as an example, dividing a unit circle into 36 parts to obtain normalized feature vectors, visualizing the normalized feature vectors by using bar graphs, and respectively representing a main direction, an auxiliary direction and a mirror image feature vector.
Five images are obtained from the retrieval result, the retrieval image is removed, at most four correct images are obtained, the algorithm retrieves two correct images, and the retrieval result is displayed in a three-dimensional image mode.
The method can eliminate the influence of the starting point on the contour retrieval, has the invariance of rotation, translation, scale and mirror image, has low feature dimension and high extraction speed, and can obtain higher retrieval rate and better universality compared with other contour retrieval algorithms.
Example 2
A computer-readable storage medium having stored thereon a computer program that executes a method of implementing the tangential domain-based three-dimensional image contour retrieval method described above.
The above embodiments are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments, and any other changes, modifications, substitutions, combinations, and simplifications which do not depart from the spirit and principle of the present invention should be construed as equivalents thereof, and all such changes, modifications, substitutions, combinations, and simplifications are intended to be included in the scope of the present invention.

Claims (9)

1. A three-dimensional image contour retrieval method based on a tangential domain is characterized by comprising the following steps:
generation of S1 projection diagram: inputting a three-dimensional image, zooming and translating the three-dimensional image to the center of a unit sphere, generating viewpoints on the unit sphere, and generating a projection drawing at the position of each viewpoint;
s2 feature view extraction: extracting the sequence contour of each projection drawing, and recording the sequence contour as CiWherein i is the number of pictures and uses a one-dimensional equation CiExpressed as x (t) + jy (t), for CiPerforming Fourier series expansion, selecting the first n Fourier coefficients for description, randomly selecting four clustering centers, clustering by using a K-means algorithm, and taking an image closest to the clustering centers as a characteristic view of the image;
s3 arc length equal resampling: counting the number of pixels of the characteristic view as the total arc length according to
Figure FDA0002860086880000011
Carrying out arc length parameterization, wherein m represents the arc length of the position of the sequence contour point, carrying out cubic B-spline interpolation and carrying out arc length equal interval resampling to obtain the sequence contour；
S4, convolving the sequence contour obtained in the last step with a Gaussian kernel function;
s5 tangential domain feature extraction: t ═ x (s, σ)max)′，y(s，σmax) '), mapping T to a unit circle, dividing the unit circle according to a certain angle interval margin, counting the number n (k) of points in each bin, wherein k is 1,2maxAnd greater than 0.8nmaxThe angles are respectively used as a main direction and an auxiliary direction, and the characteristic vector is converted to the main direction fmain(i) And an auxiliary direction fau(i) Then mirror the features fmir(i) To obtain CiIs noted as fj(j＝1，2，3，4；)，fj＝{fmain(i)，fau(i)，fmir(i) J represents the number of feature contours;
s6 definition Dpq：dist(fq 1，fp 1)＝min[sum(fp 1-fq 1)2](p≠q)，
Figure FDA0002860086880000012
Figure FDA0002860086880000013
[·]Representing and merging into a vector;
s7 defines the threshold and retrieves: will DpqAnd according to the sequence from small to large, taking the maximum value in the first four-dimensional minimum values of a certain class as a threshold value for searching the image.
2. The method for retrieving a contour of a three-dimensional image according to claim 1, wherein in S1, a viewpoint is generated on a unit sphere by using a gaussian seidel iteration.
3. The method for retrieving a contour of a three-dimensional image according to claim 1 or 2, wherein the number of the viewpoints is 200.
4. The method for retrieving a contour of a three-dimensional image according to claim 1, wherein in said S2, four feature views are generated.
5. The method for retrieving a three-dimensional image contour according to claim 1, wherein in said S3, the number of resample points is 400.
6. The method of claim 1, wherein C is assigned due to translational invariance in the tangential domain and a one-to-one mapping between the closed sequence contours and the tangential domainiConversion to the tangential domain for expression and feature extraction.
7. The method of claim 1, wherein the extracting the sequence contour of each projection image is performed by using a morphological watershed algorithm.
8. A system for implementing the three-dimensional image contour retrieval method according to any one of claims 1 to 7, comprising:
the projection graph generation module: inputting a three-dimensional image, zooming and translating the three-dimensional image to the center of a unit sphere, generating viewpoints on the unit sphere, and generating a projection drawing at the position of each viewpoint;
the characteristic view extraction module: extracting the sequence contour of each projection drawing, and obtaining a characteristic view of the projection drawing after processing;
arc length equal interval sampling module: counting the number of pixels of the characteristic view as the total arc length, and performing resampling on the arc length at equal intervals to obtain a sequence profile;
a convolution module: convolving the obtained characteristic view sequence outline with a Gaussian kernel function;
a tangential domain feature extraction module: obtaining a feature vector of the projection drawing;
the retrieval module: definition DpqAnd according to the sequence from small to large, taking the maximum value in the first four-dimensional minimum values of a certain class as a threshold value for searching the image.
9. A storage medium having stored thereon a computer program for executing the three-dimensional image contour detection method according to any one of claims 1 to 7.
CN202011563813.3A 2020-12-25 2020-12-25 Three-dimensional image contour retrieval method and system based on tangential domain and storage medium Active CN112541092B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011563813.3A CN112541092B (en) 2020-12-25 2020-12-25 Three-dimensional image contour retrieval method and system based on tangential domain and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011563813.3A CN112541092B (en) 2020-12-25 2020-12-25 Three-dimensional image contour retrieval method and system based on tangential domain and storage medium

Publications (2)

Publication Number Publication Date
CN112541092A true CN112541092A (en) 2021-03-23
CN112541092B CN112541092B (en) 2022-04-22

Family

ID=75018177

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011563813.3A Active CN112541092B (en) 2020-12-25 2020-12-25 Three-dimensional image contour retrieval method and system based on tangential domain and storage medium

Country Status (1)

Country Link
CN (1) CN112541092B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113012048A (en) * 2021-03-26 2021-06-22 电子科技大学 Conveyor belt surface object image splicing method based on SIFT features

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2008056757A1 (en) * 2006-11-10 2008-05-15 National University Corporation Toyohashi University Of Technology Three-dimensional model search method, computer program, and three-dimensional model search system
CN101593205A (en) * 2009-06-24 2009-12-02 清华大学 Method for searching three-dimension model based on video
CN103902657A (en) * 2014-03-03 2014-07-02 浙江大学 Three-dimensional model retrieval method based on sketch
CN105930497A (en) * 2016-05-06 2016-09-07 浙江工业大学 Image edge and line feature based three-dimensional model retrieval method
CN106021330A (en) * 2016-05-06 2016-10-12 浙江工业大学 A three-dimensional model retrieval method used for mixed contour line views
US20180286108A1 (en) * 2017-03-29 2018-10-04 Fujifilm Corporation Three-dimensional image processing apparatus, three-dimensional image processing method, and three-dimensional image processing program

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2008056757A1 (en) * 2006-11-10 2008-05-15 National University Corporation Toyohashi University Of Technology Three-dimensional model search method, computer program, and three-dimensional model search system
CN101593205A (en) * 2009-06-24 2009-12-02 清华大学 Method for searching three-dimension model based on video
CN103902657A (en) * 2014-03-03 2014-07-02 浙江大学 Three-dimensional model retrieval method based on sketch
CN105930497A (en) * 2016-05-06 2016-09-07 浙江工业大学 Image edge and line feature based three-dimensional model retrieval method
CN106021330A (en) * 2016-05-06 2016-10-12 浙江工业大学 A three-dimensional model retrieval method used for mixed contour line views
US20180286108A1 (en) * 2017-03-29 2018-10-04 Fujifilm Corporation Three-dimensional image processing apparatus, three-dimensional image processing method, and three-dimensional image processing program

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
刘志等: "基于特征线条的三维模型检索方法", 《计算机辅助设计与图形学学报》 *
唐祺等: "基于概率密度和轮廓的三维模型检索", 《计算机工程》 *
贺巍等: "一种基于视点距离的三维模型特征提取算法", 《小型微型计算机系统》 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113012048A (en) * 2021-03-26 2021-06-22 电子科技大学 Conveyor belt surface object image splicing method based on SIFT features

Also Published As

Publication number Publication date
CN112541092B (en) 2022-04-22

Similar Documents

Publication Publication Date Title
JP6216508B2 (en) Method for recognition and pose determination of 3D objects in 3D scenes
Le et al. Interactive facial feature localization
Xie et al. Shape matching and modeling using skeletal context
US9098770B2 (en) Image processing device and image processing method, and program
Mittal et al. Generalized projection-based M-estimator
US20140043329A1 (en) Method of augmented makeover with 3d face modeling and landmark alignment
CN108027878A (en) Method for face alignment
Fan et al. Convex hull indexed Gaussian mixture model (CH-GMM) for 3D point set registration
Nasri et al. A novel approach for dynamic hand gesture recognition using contour-based similarity images
Kim et al. Robust facial landmark extraction scheme using multiple convolutional neural networks
Baumberger et al. 3d face reconstruction from video using 3d morphable model and silhouette
CN112541092B (en) Three-dimensional image contour retrieval method and system based on tangential domain and storage medium
Chang et al. Visual tracking in high-dimensional state space by appearance-guided particle filtering
CN111709269B (en) Human hand segmentation method and device based on two-dimensional joint information in depth image
Lv et al. 3D facial similarity measurement and its application in facial organization
CN114723973A (en) Image feature matching method and device for large-scale change robustness
Lin et al. 6D object pose estimation with pairwise compatible geometric features
Chun et al. 3D star skeleton for fast human posture representation
Abeysundera et al. Nearest neighbor weighted average customization for modeling faces
CN116704587B (en) Multi-person head pose estimation method and system integrating texture information and depth information
Mohanna et al. An efficient active contour model through curvature scale space filtering
Wei et al. BuilDiff: 3D Building Shape Generation using Single-Image Conditional Point Cloud Diffusion Models
CN113570667B (en) Visual inertial navigation compensation method and device and storage medium
GAO et al. Face Reconstruction Algorithm based on Lightweight Convolutional Neural Networks and Channel-wise Attention
Ravikumar Lightweight markerless monocular face capture with 3d spatial priors

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant