CN110084834B - Target tracking method based on rapid tensor singular value decomposition feature dimension reduction - Google Patents

Target tracking method based on rapid tensor singular value decomposition feature dimension reduction Download PDF

Info

Publication number
CN110084834B
CN110084834B CN201910349128.1A CN201910349128A CN110084834B CN 110084834 B CN110084834 B CN 110084834B CN 201910349128 A CN201910349128 A CN 201910349128A CN 110084834 B CN110084834 B CN 110084834B
Authority
CN
China
Prior art keywords
tensor
feature
dimension reduction
features
value decomposition
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910349128.1A
Other languages
Chinese (zh)
Other versions
CN110084834A (en
Inventor
傅衡成
周武能
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Donghua University
Original Assignee
Donghua University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Donghua University filed Critical Donghua University
Priority to CN201910349128.1A priority Critical patent/CN110084834B/en
Publication of CN110084834A publication Critical patent/CN110084834A/en
Application granted granted Critical
Publication of CN110084834B publication Critical patent/CN110084834B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20048—Transform domain processing
    • G06T2207/20056—Discrete and fast Fourier transform, [DFT, FFT]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a target tracking method based on rapid tensor singular value decomposition feature dimension reduction, which comprises the following steps: extracting various characteristics of each frame of video data and constructing a tensor structure; performing singular value decomposition on the constructed tensor; and training a relevant filter for the features after dimension reduction, and tracking the target. The method can effectively reduce the number of the features and accelerate the tracking speed, and compared with the traditional vector-based principal component analysis feature dimension reduction mode and the like, the method better retains the structural information of the features; tensor singular value decomposition has invariance to the rotation of features to enhance the robustness of the tracker to the rotation of the target.

Description

Target tracking method based on rapid tensor singular value decomposition feature dimension reduction
Technical Field
The invention relates to a target tracking method based on rapid tensor singular value decomposition feature dimension reduction, and belongs to the technical field of video target tracking.
Background
The target tracking has important significance for the development of the fields of robots, unmanned planes, automatic driving, navigation, guidance and the like. For example, in the human-computer interaction process, the camera continuously tracks the human behavior, and the robot achieves the understanding of the human posture, the human motion and the human gesture through a series of analysis processing, so that the friendly communication between the human and the machine is better realized; in the unmanned aerial vehicle target tracking process, visual information of a target is continuously acquired and transmitted to a ground control station, and a video image sequence is analyzed through an algorithm to obtain real-time position information of the tracked target so as to ensure that the tracked target is within the visual field range of the unmanned aerial vehicle in real time.
In recent years, a related filtering type target tracking method not only has higher tracking speed but also has good tracking precision, but also has seriously reduced tracking speed of related filters along with the continuous increase of various characteristics.
In recent years, image features used for correlation filtering, such as color name features, gradient histogram features, and depth features of a depth convolution neural network, have increased tracking accuracy, but have decreased tracking speed of the correlation filter.
Disclosure of Invention
The purpose of the invention is: provided is a target tracking method which can improve the accuracy by using more characteristics and reduce the loss of speed.
In order to achieve the above object, the technical solution of the present invention is to provide a target tracking method based on fast tensor singular value decomposition feature dimension reduction, which is characterized in that a tensor singular value decomposition feature dimension reduction feature is used for target tracking, and the method includes the following steps:
(1) extracting gradient direction histogram features HOG, color name features CN and pre-trained depth convolution features CNN of a t frame tracking result window;
(2) discharging the features extracted in the step (1) into a horizontal section of a tensor to form 4 mutually independent third-order tensors which are respectively recorded as Li，i＝1,2,3,4；
(3) Respectively calculating average characteristics by taking the horizontal slice of each tensor as a unit, and recording the average characteristics of the ith characteristic tensor as MiThen, there are:
Figure BDA0002043343650000021
in the formula, NiThe number of horizontal slices for the ith feature tensor; l isi(j,: means a third order tensor LiThe jth horizontal slice of (a);
(4) the horizontal slice of each feature tensor is subtracted by the corresponding average feature and is recorded as Ai；
(5) Transforming the time-domain feature tensor into the frequency domain using a fast Fourier transform, and for each feature in the frequency domainPerforming traditional matrix singular value decomposition on each horizontal slice of the feature tensor, and intercepting columns of a left singular matrix, wherein the reserved number is that the front k dimension of the columns is also equal to the dimension of the feature tensor after dimension reduction; after the singular value decomposition of each side slice is completed, the left singular matrixes which are reserved are formed into left singular eigenvectors according to the original arrangement sequence, finally, the left singular eigenvectors are converted into time domains from the frequency domains through the inverse fast Fourier transform, and the left singular eigenvectors of each time domain form are respectively recorded as the left singular eigenvectors
Figure BDA0002043343650000026
(6) Carrying out tensor product operation by utilizing the left singular characteristic tensor of the time domain form and the characteristic tensor which subtracts the average characteristic to obtain a characteristic tensor after dimension reduction, and adding the average characteristic which is obtained correspondingly before to each front slice of each characteristic tensor after dimension reduction to obtain a characteristic tensor Fi：
Figure BDA0002043343650000022
In the formula, tprod (·) represents a tensor product; tran (U)i) Transpose of the representation of the feature tensor U
(7) For each feature tensor FiTransposing, placing the front slices into the side slices, and then applying each feature tensor FiThe side panels of (a) are all trained and noted as a filter
Figure BDA0002043343650000024
And updating the previous filter, wherein the updated formula is shown as the following formula:
Figure BDA0002043343650000023
in the formula (I), the compound is shown in the specification,
Figure BDA0002043343650000025
a filter representing the ith feature of the t-th frame, wherein eta is the learning rate of the filter;
(8) extracting gradient direction histogram features HOG, color name features CN and depth convolution features CNN of the candidate region of the t +1 th frame, keeping the discharging sequence consistent with the step (2), and carrying out tensor multiplication on a projection operator obtained by the t-th frame and each feature tensor of the t +1 th frame to obtain a feature tensor after dimension reduction;
(9) performing convolution operation on the feature tensor subjected to dimensionality reduction by using the filter obtained from the t-th frame, obtaining a confidence map by slicing the side surface of each feature tensor, adding the confidence maps to obtain a response map, and taking the position with the maximum response map as the position of a target in the t + 1-th frame;
(10) and (4) judging whether the frame is the last frame or not, if not, making t equal to t +1, returning to the step (1), and if so, stopping tracking.
Preferably, in step (1), the histogram of gradient direction feature HOG includes 31 layers, the color name feature CN includes 11 layers, the Layer1 of the depth convolution feature CNN includes 96 layers, and the Layer5 includes 512 layers.
Preferably, in step (5), the matrix singular value decomposition in the frequency domain form is as follows, and the projection operator is updated according to the following formula:
Figure BDA0002043343650000031
in the formula, Pi tAnd a represents the learning rate of the projection operator.
The method can effectively reduce the number of the features and accelerate the tracking speed, and compared with the traditional vector-based principal component analysis feature dimension reduction mode and the like, the method better retains the structural information of the features; tensor singular value decomposition has invariance to the rotation of features to enhance the robustness of the tracker to the rotation of the target.
Drawings
FIG. 1 is a flow algorithm implemented by the present invention;
figure 2 is an example of tensor feature dimension reduction as practiced by the present invention.
Detailed Description
The invention will be further illustrated with reference to the following specific examples. It should be understood that these examples are for illustrative purposes only and are not intended to limit the scope of the present invention. Further, it should be understood that various changes or modifications of the present invention may be made by those skilled in the art after reading the teaching of the present invention, and such equivalents may fall within the scope of the present invention as defined in the appended claims.
The invention provides a target tracking method based on rapid tensor singular value decomposition feature dimension reduction, which comprises the following steps of:
(1) and extracting gradient direction histogram features HOG, color name features CN and pre-trained depth convolution features CNN of the t frame tracking result window. The histogram of gradient direction feature HOG includes 31 layers, the color name feature CN includes 11 layers, the Layer1 of the depth convolution feature CNN includes 96 layers, and the Layer5 includes 512 layers.
(2) The features are arranged into horizontal slices of tensors to form 4 mutually independent third-order tensors which are respectively marked as Li，i＝1,2,3,4。
(3) The average features are calculated in units of horizontal slices per tensor, respectively.
Figure BDA0002043343650000041
In the formula, NiThe number of horizontal slices of the ith feature tensor is 31, for example, the number of horizontal slices of the HOG feature tensor is 31; l isi(j,: means a third order tensor LiThe jth horizontal slice of (1), MiIs the average characteristic of the ith tensor.
(4) The horizontal slice of each feature tensor is subtracted by the corresponding average feature and is recorded as AiWherein i is 1, 2, 3, 4.
(5) Converting the time domain feature tensor into the frequency domain using a fast Fourier transform, and performing a conventional moment on each horizontal slice of each feature tensor in the frequency domainAnd performing matrix singular value decomposition, and intercepting columns of a left singular matrix, wherein the reserved front k dimensions of the columns are also equal to the dimensions of the feature tensor after dimension reduction. After the singular value decomposition of each side slice is completed, the reserved left singular matrixes are formed into tensors according to the original arrangement sequence, finally, the left singular eigenvector is converted from the frequency domain to the time domain through the inverse fast Fourier transform, and the left singular eigenvector (also called as a projection operator) of each time domain form is recorded as a projection operator
Figure BDA0002043343650000042
The matrix singular value decomposition in the form of the frequency domain is shown as the following formula, and the projection operator is updated by the following formula:
Figure BDA0002043343650000043
in the formula, Pi tA projection operator representing the ith feature tensor for the tth frame; α represents the learning rate of the projection operator.
(6) And carrying out tensor product operation by utilizing the left singular tensor of the time domain form and the feature tensor of which the average feature is subtracted to obtain a feature tensor after dimension reduction, and adding the previously obtained average feature to each front slice of each feature tensor after dimension reduction. As shown in the following formula:
Figure BDA0002043343650000051
in the formula, tprod (·) represents the tensor product, tran (U)i) Representing tensor UiTranspose of (F)iRepresenting the resulting feature tensor.
(7) Transposing each feature tensor, arranging the front slices to the side slices, then training a filter for each tensor side slice and recording as
Figure BDA0002043343650000052
And performing a further operation on the previous filterThe new, updated formula is shown as follows:
Figure BDA0002043343650000053
in the formula (I), the compound is shown in the specification,
Figure BDA0002043343650000054
and eta is the learning rate of the filter.
(8) And (3) extracting gradient direction histogram features HOG, color name features CN and depth convolution features CNN of the candidate region of the t +1 th frame, keeping the discharging sequence consistent with the step (2), and carrying out tensor multiplication on the projection operator obtained by the t th frame and each feature tensor of the t +1 th frame to obtain the feature tensor after dimension reduction.
(9) And performing convolution operation on the feature tensor after dimensionality reduction by using the filter obtained from the t-th frame, obtaining a confidence map by side slicing of each feature tensor, adding the confidence maps to obtain a response map, and taking the position with the maximum response map as the position of the target in the t + 1-th frame.
(10) And (4) judging whether the frame is the last frame or not, if not, making t equal to t +1, returning to the step (1), and if so, stopping tracking.

Claims (3)

1. A target tracking method based on fast tensor singular value decomposition feature dimension reduction is characterized in that the tensor singular value decomposition feature dimension reduction feature is utilized to track a target, and the method comprises the following steps:
(1) extracting gradient direction histogram features HOG, color name features CN and pre-trained depth convolution features CNN of a t frame tracking result window;
(2) discharging the features extracted in the step (1) into a horizontal section of a tensor to form 4 mutually independent third-order tensors which are respectively recorded as Li，i＝1,2,3,4；
(3) Respectively calculating average characteristics by taking the horizontal slice of each tensor as a unit, and recording the average characteristics of the ith characteristic tensor as MiThen, there are:
Figure FDA0002825435570000011
in the formula, NiThe number of horizontal slices for the ith feature tensor; l isi(j,: means a third order tensor LiThe jth horizontal slice of (a);
(4) the horizontal slice of each feature tensor is subtracted by the corresponding average feature and is recorded as Ai；
(5) Converting a time domain feature tensor into a frequency domain by utilizing fast Fourier transform, performing traditional matrix singular value decomposition on each horizontal slice of each feature tensor in the form of the frequency domain, and intercepting columns of a left singular matrix, wherein the reserved front k dimensions of the columns are also equal to the dimensions of the feature tensor after dimension reduction; after the singular value decomposition of each side slice is completed, the left singular matrixes which are reserved are formed into left singular eigenvectors according to the original arrangement sequence, finally, the left singular eigenvectors are converted into time domains from the frequency domains through the inverse fast Fourier transform, and the left singular eigenvectors of each time domain form are respectively recorded as the left singular eigenvectors
Figure FDA0002825435570000012
(6) Carrying out tensor product operation by utilizing the left singular characteristic tensor of the time domain form and the characteristic tensor which subtracts the average characteristic to obtain a characteristic tensor after dimension reduction, and adding the average characteristic which is obtained correspondingly before to each front slice of each characteristic tensor after dimension reduction to obtain a characteristic tensor Fi：
Figure FDA0002825435570000013
In the formula, tprod (·) represents a tensor product; tran (U)i) Tensor U representing featuresiIs transferred to
(7) For each feature tensor FiTransposing, arranging the front slicesTo the side slice, and then for each feature tensor FiThe side panels of (a) are all trained and noted as a filter
Figure FDA0002825435570000021
And updating the previous filter, wherein the updated formula is shown as the following formula:
Figure FDA0002825435570000022
in the formula (I), the compound is shown in the specification,
Figure FDA0002825435570000023
a filter representing the ith feature of the t-th frame, wherein eta is the learning rate of the filter;
(8) extracting gradient direction histogram features HOG, color name features CN and depth convolution features CNN of the candidate region of the t +1 th frame, keeping the discharging sequence consistent with the step (2), and carrying out tensor multiplication on a projection operator obtained by the t-th frame and each feature tensor of the t +1 th frame to obtain a feature tensor after dimension reduction;
(9) performing convolution operation on the feature tensor subjected to dimensionality reduction by using the filter obtained from the t-th frame, obtaining a confidence map by slicing the side surface of each feature tensor, adding the confidence maps to obtain a response map, and taking the position with the maximum response map as the position of a target in the t + 1-th frame;
(10) and (4) judging whether the frame is the last frame or not, if not, making t equal to t +1, returning to the step (1), and if so, stopping tracking.
2. The fast tensor singular value decomposition feature dimension reduction-based target tracking method as claimed in claim 1, wherein in step (1), the gradient direction histogram feature HOG comprises 31 layers, the color name feature CN comprises 11 layers, the Layer1 of the deep convolution feature CNN comprises 96 layers, and the Layer5 comprises 512 layers.
3. The method for tracking the target based on the feature dimension reduction of the fast tensor singular value decomposition as claimed in claim 1, wherein in the step (5), the matrix singular value decomposition in the frequency domain form is as follows, and the projection operator is updated according to the following formula:
Figure FDA0002825435570000024
in the formula, Pi tAnd a represents the learning rate of the projection operator.
CN201910349128.1A 2019-04-28 2019-04-28 Target tracking method based on rapid tensor singular value decomposition feature dimension reduction Active CN110084834B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910349128.1A CN110084834B (en) 2019-04-28 2019-04-28 Target tracking method based on rapid tensor singular value decomposition feature dimension reduction

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910349128.1A CN110084834B (en) 2019-04-28 2019-04-28 Target tracking method based on rapid tensor singular value decomposition feature dimension reduction

Publications (2)

Publication Number Publication Date
CN110084834A CN110084834A (en) 2019-08-02
CN110084834B true CN110084834B (en) 2021-04-06

Family

ID=67417289

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910349128.1A Active CN110084834B (en) 2019-04-28 2019-04-28 Target tracking method based on rapid tensor singular value decomposition feature dimension reduction

Country Status (1)

Country Link
CN (1) CN110084834B (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110766723B (en) * 2019-10-22 2020-11-24 湖南大学 Unmanned aerial vehicle target tracking method and system based on color histogram similarity
CN116202760B (en) * 2023-05-05 2023-08-18 赛腾机电科技（常州）有限公司 Singular value decomposition method and system for third-order tensor for mechanical fault diagnosis
CN117058886A (en) * 2023-10-12 2023-11-14 安徽宇疆科技有限公司 Beidou space-time data model based on third-order tensor and traffic flow analysis method

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0920120A3 (en) * 1997-11-27 2001-08-29 Vectron Elektronik GmbH Method and apparatus for controlling the movement path of the workpiece supporting head of an orbital vibration welding system
CN102592135A (en) * 2011-12-16 2012-07-18 温州大学 Visual tracking method of subspace fusing target space distribution and time sequence distribution characteristics
CN104751169A (en) * 2015-01-10 2015-07-01 哈尔滨工业大学（威海） Method for classifying rail failures of high-speed rail
CN107093189A (en) * 2017-04-18 2017-08-25 山东大学 Method for tracking target and system based on adaptive color feature and space-time context
CN108305297A (en) * 2017-12-22 2018-07-20 上海交通大学 A kind of image processing method based on multidimensional tensor dictionary learning algorithm
US10213274B1 (en) * 2018-03-12 2019-02-26 King Saud University Method of tracking and navigation for a dental instrument
CN109447073A (en) * 2018-11-08 2019-03-08 电子科技大学 A kind of method for detecting infrared puniness target based on tensor Robust Principal Component Analysis

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104021395B (en) * 2014-06-20 2017-05-03 华侨大学 Target tracing algorithm based on high-order partial least square method
CN108665481B (en) * 2018-03-27 2022-05-31 西安电子科技大学 Self-adaptive anti-blocking infrared target tracking method based on multi-layer depth feature fusion
CN108764249B (en) * 2018-04-23 2021-07-09 云南民族大学 Rotation-invariant local feature description method, system and device for multi-source image

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0920120A3 (en) * 1997-11-27 2001-08-29 Vectron Elektronik GmbH Method and apparatus for controlling the movement path of the workpiece supporting head of an orbital vibration welding system
CN102592135A (en) * 2011-12-16 2012-07-18 温州大学 Visual tracking method of subspace fusing target space distribution and time sequence distribution characteristics
CN104751169A (en) * 2015-01-10 2015-07-01 哈尔滨工业大学（威海） Method for classifying rail failures of high-speed rail
CN107093189A (en) * 2017-04-18 2017-08-25 山东大学 Method for tracking target and system based on adaptive color feature and space-time context
CN108305297A (en) * 2017-12-22 2018-07-20 上海交通大学 A kind of image processing method based on multidimensional tensor dictionary learning algorithm
US10213274B1 (en) * 2018-03-12 2019-02-26 King Saud University Method of tracking and navigation for a dental instrument
CN109447073A (en) * 2018-11-08 2019-03-08 电子科技大学 A kind of method for detecting infrared puniness target based on tensor Robust Principal Component Analysis

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
《Computation Estimation of Scene Structure Through Texture Gradient Cues》;Christopher W et al;《Electronic Imaging》;20170129;全文 *
《张量核范数回归的目标跟踪》;亚森江.木沙等;《中国图象图形学报》;20160713;第21卷(第6期);全文 *
张亚红.《 基于显著性检测和压缩感知的视觉跟踪》.《中国优秀硕士学位论文全文数据库信息科技辑》.2016,(第2016年第10期),第I138-364页. *

Also Published As

Publication number Publication date
CN110084834A (en) 2019-08-02

Similar Documents

Publication Publication Date Title
CN110210551B (en) Visual target tracking method based on adaptive subject sensitivity
CN110084834B (en) Target tracking method based on rapid tensor singular value decomposition feature dimension reduction
CN104574445B (en) A kind of method for tracking target
CN112184752A (en) Video target tracking method based on pyramid convolution
CN110084836B (en) Target tracking method based on deep convolution characteristic hierarchical response fusion
CN112348849B (en) Twin network video target tracking method and device
CN107016689A (en) A kind of correlation filtering of dimension self-adaption liquidates method for tracking target
CN107154024A (en) Dimension self-adaption method for tracking target based on depth characteristic core correlation filter
CN109859241B (en) Adaptive feature selection and time consistency robust correlation filtering visual tracking method
CN111311647B (en) Global-local and Kalman filtering-based target tracking method and device
WO2021253686A1 (en) Feature point tracking training and tracking methods, apparatus, electronic device, and storage medium
CN113361636B (en) Image classification method, system, medium and electronic device
CN112734809B (en) On-line multi-pedestrian tracking method and device based on Deep-Sort tracking framework
CN111724411B (en) Multi-feature fusion tracking method based on opposite-impact algorithm
CN107301382B (en) Behavior identification method based on deep nonnegative matrix factorization under time dependence constraint
CN111079547B (en) Pedestrian moving direction identification method based on mobile phone inertial sensor
CN109858454B (en) Adaptive kernel correlation filtering tracking method based on dual models
CN113643329B (en) Twin attention network-based online update target tracking method and system
CN110189362B (en) Efficient target tracking method based on multi-branch self-coding countermeasure network
CN112258557B (en) Visual tracking method based on space attention feature aggregation
CN109727272B (en) Target tracking method based on double-branch space-time regularization correlation filter
CN111582349A (en) Improved target tracking algorithm based on YOLOv3 and kernel correlation filtering
CN107798329B (en) CNN-based adaptive particle filter target tracking method
CN114283402A (en) License plate detection method based on knowledge distillation training and space-time combined attention
CN111862167A (en) Rapid robust target tracking method based on sparse compact correlation filter

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant