CN113763427B - Multi-target tracking method based on coarse-to-fine shielding processing - Google Patents

Multi-target tracking method based on coarse-to-fine shielding processing Download PDF

Info

Publication number
CN113763427B
CN113763427B CN202111035065.6A CN202111035065A CN113763427B CN 113763427 B CN113763427 B CN 113763427B CN 202111035065 A CN202111035065 A CN 202111035065A CN 113763427 B CN113763427 B CN 113763427B
Authority
CN
China
Prior art keywords
shielding
pedestrian
target
model
pedestrians
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202111035065.6A
Other languages
Chinese (zh)
Other versions
CN113763427A (en
Inventor
路小波
张帅帅
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Southeast University
Original Assignee
Southeast University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Southeast University filed Critical Southeast University
Priority to CN202111035065.6A priority Critical patent/CN113763427B/en
Publication of CN113763427A publication Critical patent/CN113763427A/en
Application granted granted Critical
Publication of CN113763427B publication Critical patent/CN113763427B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/20Analysis of motion
    • G06T7/246Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/21Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/24Classification techniques
    • G06F18/241Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2411Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10016Video; Image sequence
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02TCLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00Road transport of goods or passengers
    • Y02T10/10Internal combustion engine [ICE] based vehicles
    • Y02T10/40Engine management systems

Abstract

The invention discloses a multi-target tracking method based on coarse-to-fine shielding treatment, which comprises the following steps of: firstly, adding an occlusion score prediction on the basis of a JDE model pre-measurement head, thereby completing the complete processing of a non-occlusion target and the rough processing of the occlusion target; on the basis, training the mapped and cut area for shielding pedestrians as a training set of the second step model, and finishing accurate detection and apparent feature vector extraction for shielding pedestrians; thereby realizing the fine treatment of the shielding target; and integrating the results output by the two-step model, and completing the tracking of pedestrians by using a data association algorithm. The invention solves the problem that the prior art can not accurately track pedestrians in a scene with shielding condition, and can be well adapted to public environments with various time periods and various pedestrian densities; the pedestrian tracking method has a good effect on pedestrian tracking.

Description

Multi-target tracking method based on coarse-to-fine shielding processing
Technical Field
The invention belongs to the field of computer vision and monitoring video analysis, and particularly relates to a multi-target tracking method based on coarse-to-fine shielding processing.
Background
Multi-objective tracking is an important component of surveillance video analysis. The method can be directly used for analyzing the motion trail of the object, and can be used as a research basis for advanced tasks such as object motion recognition, behavior analysis and the like.
In order to accomplish the multi-objective tracking task, many mainstream deep learning algorithms propose strategies for tracking based on detection. These methods divide multi-target tracking into a detection module and an embedding module. The detection module completes target detection, the embedding module extracts the characteristics of the target by using a related algorithm, however, repeated calculation can occur between the two modules for a plurality of times, and the running speed is influenced. For this reason, some scientists have proposed a method of integrating a detection module and an embedding module into one neural network, the two modules sharing the same underlying characteristics, thereby avoiding repeated calculations and improving performance. However, due to limitations of the self-detection framework, these methods do not perform well in some scenarios for detection and tracking of occluding objects. In particular, the detection framework often detects two occluded objects as one object, which also presents some problems for object tracking. For occlusion target detection problems, some improved non-maximum suppression algorithms are proposed, such as soft-NMS, adaptive-NMS, etc., and some improved loss functions are proposed, such as retransmission loss, etc., but these methods all depend on the predicted results of the original detection network, and for some scenarios, the occlusion detection is not good.
Disclosure of Invention
In order to solve the problems, the invention discloses a multi-target tracking method based on coarse-to-fine shielding treatment, which separates the treatment of a non-shielding target from a shielding target, and in a first step model, focuses on the non-shielding target and positions the shielding target to realize the coarse treatment of the shielding target; and the second step of model realizes the fine treatment of the shielding target, thereby realizing the improvement of the pedestrian tracking performance in some shielding environments.
In order to achieve the above purpose, the technical scheme of the invention is as follows:
a multi-target tracking method based on coarse-to-fine occlusion processing, comprising the steps of:
firstly, labeling the data set obtained from public places, constructing a training set and a testing set for pedestrian tracking,
training a first step model of the multi-target tracking method by utilizing part of information marked by the training set to realize detection of non-shielding pedestrian targets, extraction of corresponding apparent feature vectors and positioning of shielding pedestrian areas, thereby completing complete processing of the non-shielding targets and rough processing of the shielding targets,
and thirdly, mapping the area for shielding the pedestrians on the basis of the training completion of the first step model, cutting out the area as a training set of the second step model for training, and completing detection and apparent feature vector extraction of the shielded pedestrians. Thereby realizing the fine treatment of the shielding target,
and step four, integrating the results output by the two-step model, and completing the tracking of pedestrians by using a data association algorithm.
In the first step, the pedestrian tracking dataset is constructed on the monitoring videos shot by the cameras in the public places, the labeling content comprises a pedestrian bounding box id (marked as 0 when no shielding condition exists and increased id when 2 persons are shielded mutually), the pedestrian id (marked as-1 when shielding condition exists and increased from 0 otherwise), the bounding box position, whether shielding exists for the pedestrian in the bounding box (shielding mark 1 and non-shielding mark 0), and the pedestrian target is contained in which bounding box (marked as 0 when shielding condition exists and marked as shielding bounding box id containing the pedestrian target when no shielding condition exists);
further, in the second step, the first step model of the method defines two kinds of bounding box regression, one is that of a single pedestrian without shielding, and the other is that of two pedestrians with shielding; on the basis of the JDE model, the first step model adds shielding fraction prediction in a prediction head so that the prediction head can judge whether a pedestrian in a regressed bounding box is shielded or not; if no shielding exists, the position and the apparent feature vector of the pedestrian can be extracted at the same time, and if the shielding exists, the shielding area is positioned, so that the rough treatment of shielding the pedestrian is completed.
In the third step, the occlusion region located in the second step is mapped to the small-scale feature map of the first step model, and is cut by using an ROI alignment algorithm as an input of the second step model, the second step model performs fine processing on the occlusion region feature map to obtain a position and an apparent feature vector of an occluded pedestrian, wherein when the boundary box of the occluded pedestrian is trained, a loss function adopts a mode of combining SmoothL1 loss and Repgt and Repbox loss weighting, and the boundary box loss function is calculated as follows:
wherein,representing smoothL1 loss function, L RepGT Representing the RepGT loss function, L RepBox Indicating the RepBox loss function, and α, β indicating the weighting parameters.
Further, in the fourth step, the results in the first step and the second step are combined to obtain the positions and apparent feature vectors of all the pedestrian targets in the current frame, and the characteristics of similarity and small position variation of the same pedestrian target apparent feature vector between adjacent frames are utilized to complete the matching of the pedestrian targets, so that the tracking of the multi-target pedestrians is finally completed.
The beneficial effects of the invention are as follows:
1) The invention creatively provides a coarse-to-fine shielding processing method, which comprises the steps of firstly completing coarse processing of shielding targets and complete processing of non-shielding targets, secondly completing fine processing of shielding targets, and finally comprehensively obtaining final pedestrian positions and corresponding apparent feature vectors by the results of the two steps.
2) According to the first-step model, based on the JDE model, the shielding score prediction is added in the prediction head, so that whether the shielding condition exists in the bounding box output by the first-step model can be judged, and if the shielding condition exists, the shielding bounding box can be positioned, and the rough processing of the shielding target is realized.
3) The invention maps the shielding pedestrian boundary box positioned by the first step model onto the small-scale feature map of the first step model (the small-scale feature map retains more information) and adopts the ROI alignment algorithm to cut, and the shielding pedestrian boundary box is used as the input of the shielding fine processing model (the second step model).
4) The invention designs a network structure of the shielding fine processing model (a second step model), and adopts a mode of combining SmoothL1 loss with RepGT and RepBox loss weighting as a loss function of shielding fine processing model boundary frame regression.
5) The invention provides a two-stage model training method aiming at a two-stage model framework, wherein in the first stage, parameters of a second-stage model are frozen, and only a first-stage model is trained; and in the second stage, on the basis of the completion of the training of the first step model, freezing parameters of the first model, and training by taking a feature map corresponding to the shielding pedestrian boundary frame positioned by the first step model as the input of the second step model. The problem that pedestrians in a scene with shielding conditions cannot be accurately tracked in the prior art is solved, and the system is well suitable for public environments with various periods and various pedestrian densities; the pedestrian tracking method has a good effect on pedestrian tracking.
Drawings
Fig. 1 is a flow chart of the present invention.
Fig. 2 is a scene diagram under a subway monitor video.
FIG. 3 is a diagram of a multi-target tracking network model framework based on coarse-to-fine occlusion processing.
FIG. 4 is a schematic diagram of a first step model predictive head and loss function.
Fig. 5 is a network configuration diagram of the occlusion finishing model (second-step model).
Fig. 6 is a multi-target tracking effect display diagram.
Detailed Description
The present invention is further illustrated in the following drawings and detailed description, which are to be understood as being merely illustrative of the invention and not limiting the scope of the invention.
Examples
The monitoring video used in the model training of this embodiment is a monitoring video in an actual subway scene, and a scene diagram is shown in fig. 2.
In this embodiment, taking the subway station monitoring video shown in fig. 2 as an example, these video images include both pedestrians in the case where there is no shielding and pedestrians in the case where there is shielding. After the video in the subway scene is obtained, marking pedestrians in the video, and thus obtaining a subway pedestrian multi-target tracking data set.
The embodiment provides a multi-target tracking method based on coarse-to-fine shielding processing, which is to perform coarse-to-fine model processing on shielding targets in an input image, and achieve better multi-target tracking effect by combining modes such as optimizing a loss function, optimizing NMS (network management system) and the like on the basis, wherein a frame diagram of the model is shown in figure 3, and the specific steps are as follows:
1) Image input processing: to eliminate the adverse effect of the input image size on model training and to take into account factors such as resolution, each frame of image in the video is resized to 1088 x 608 before being input to the model.
2) Adding occlusion score prediction: the first step model adds shielding fraction prediction into the pre-measurement head based on the JDE model, and the pre-measurement head comprises four prediction values of confidence prediction, bounding box regression prediction, shielding fraction prediction and apparent feature vector; the confidence level, the shielding score and the apparent feature vector are trained by cross entropy loss, and the boundary box regression is trained by smoothL1 loss. The weighting parameters of the loss values of all parts in the loss function are determined in an adaptive training mode. The loss function calculation formula is as follows:
wherein M is the number of the pre-measuring heads,representing confidence loss, occlusion prediction loss, bounding box regression loss and apparent feature vector loss, respectively,/>For the weighted parameters associated with each penalty, a learnable parameter is modeled. The schematic diagram is shown in fig. 4:
3) Feature map mapping and clipping: after the first step model locates the shielding boundary frame, mapping the shielding boundary frame after non-maximum suppression to a small-scale feature map of the first step model, cutting by adopting a ROIAlign algorithm, and taking the cut feature map as the input of the second step model.
4) Fine treatment of shielding targets: the purpose of the occlusion finishing model (second step model) is to accurately acquire the bounding box and corresponding apparent feature vector of the occluding pedestrian. The network structure is shown in fig. 5, the cut feature map has two branches after passing through a convolution block, one branch is used for confidence prediction and regression of a pedestrian boundary box, and the other branch is used for extracting apparent feature vectors corresponding to the blocked pedestrians.
5) Loss function of occlusion finishing model: confidence and apparent feature vector training of the occlusion finishing model still adopts cross entropy loss, and training of bounding box regression adopts a mode of combining SmoothL1 loss with RepGT and RepBox loss weighting. The purpose of SmoothL1 loss is to make the bounding box predicted value as close to the true value as possible, and the design of the repagt loss is to keep the predicted bounding box as far away as possible from its neighboring true bounding box. The purpose of the RepBox penalty is to keep the two bounding box predictors returning to different bounding boxes as far apart as possible. These three bounding box losses coordinate with each other, having a positive effect on regression of the bounding box that obscures the pedestrian. Meanwhile, the bounding box regresses the loss, the confidence coefficient is lost, and the weighting mode of the apparent feature vector loss is the same as that of the loss value of the first step model.
6) Integration and optimization: for the bounding box output by the second step model (occlusion finishing model), firstly, removing some bounding boxes which do not meet the requirements, such as the fact that the area is too small, and the coordinates of the bounding boxes exceed the image area; meanwhile, after non-maximum suppression processing is carried out on the bounding box output by the second step model by using the Soft-NMS algorithm, the bounding box output by the first step model is integrated, and then the final result is obtained by carrying out Soft-NMS algorithm processing again.
7) Data association and trajectory generation: the method comprises the steps of firstly taking cosine distance between an apparent characteristic vector group of a detection target group and an apparent characteristic vector group of a tracking target group as a main matching principle, taking distance between a boundary frame of the detection target group and a boundary frame predicted by a Kalman filter for the tracking target group as an auxiliary matching principle, respectively selecting weighting ratios of 0.95 and 0.05, and adopting a Hungary matching algorithm to finish first-step matching; secondly, in the second step of matching, the intersection ratio (IOU) of the boundary box of the detection target group and the boundary box of the tracking target group is used as a matching principle, and still the Hungary algorithm is adopted to complete the matching. For the previously unmatched tracking target marked as lost, the lost tracking target remains engaged in the matching of the next frame, and when the lost tracking target has not been matched to a new target for 25 consecutive frames (herein, frame rate is a threshold herein), this tracking target is removed from the population of tracking targets and no longer engaged in the matching of the tracking target with the detected target.
8) Two-stage model training: freezing the parameters of the second step model when the first step model is trained, and only training the first step model; after the training of the first step model is completed, the parameters of the first step model are frozen, and the feature map corresponding to the boundary box of the blocked pedestrian positioned by the first step model is used as the input of the second step model for training.
After the model is built and trained, the pedestrian in the actual monitoring video can be tracked, the tracking effect is shown in fig. 6, and the interval between two adjacent pictures is 25 frames.
The invention provides a multi-target tracking method based on coarse-to-fine shielding processing. The model is divided into two steps, wherein the first step model is used for realizing the full processing (positioning) of the non-shielding target and the rough processing (positioning) of the shielding target; and the second step of model is used for accurately positioning the blocked pedestrians and extracting apparent feature vectors thereof, so that the precise treatment of the blocking target is realized. And finally, integrating and optimizing the output results of the two models, and then carrying out data association and track generation to finally finish the multi-target tracking task. The invention has important effect on the monitoring video analysis task in the field of computer vision and has a large application prospect.
The present invention is not limited to the specific technical solutions described in the above embodiments, and other embodiments may be used in addition to the above embodiments. Any modifications, equivalent substitutions, improvements, etc. made by those skilled in the art, which are within the spirit and principles of the present invention, are intended to be included within the scope of the present invention.

Claims (3)

1. A multi-target tracking method based on coarse-to-fine occlusion processing, the method comprising the steps of:
firstly, marking a data set obtained from a public place, and constructing a training set and a testing set for pedestrian tracking;
training a first step model of the multi-target tracking method by utilizing part of information marked by the training set; the detection of non-shielding pedestrian targets and the extraction of corresponding apparent feature vectors are realized, and the positioning of shielding pedestrian areas is realized; thus completing the complete processing of the non-shielding target and the rough processing of the shielding target;
the first step model of the method defines two kinds of bounding box regression, one is that of single pedestrian without shielding, and the other is that of two pedestrians with shielding; on the basis of the JDE model, the first step model adds shielding fraction prediction in a prediction head so that the prediction head can judge whether a pedestrian in a regressed bounding box is shielded or not; if no shielding exists, the position and the apparent feature vector of the pedestrian can be extracted at the same time, and if the shielding exists, a shielding area is positioned, so that the rough treatment of shielding the pedestrian is completed;
training the mapped and cut area for shielding pedestrians as a training set of a second step model on the basis of the model training, and finishing accurate detection and apparent feature vector extraction for shielding pedestrians; thereby realizing the fine treatment of the shielding target;
mapping the occlusion region positioned in the second step into a small-scale feature map of the first step model, cutting by adopting an ROI alignment algorithm to serve as input of the second step model, and performing fine processing on the occlusion region feature map by the second step model to obtain the position and apparent feature vector of the occluded pedestrian, wherein when the boundary box of the occluded pedestrian is trained, a loss function adopts a mode of combining smoothL1 loss with RepGT and RepBox loss weighting, and the boundary box loss function is calculated as follows:
wherein,representing smoothL1 loss function, L RepGT Representing the RepGT loss function, L RepBox Representing the RepBox loss function, alpha, beta representing the weighting parameters;
and step four, integrating the results output by the two-step model, and completing the tracking of pedestrians by using a data association algorithm.
2. A multi-target tracking method based on coarse-to-fine occlusion processing as claimed in claim 1, wherein: in the first step, a pedestrian tracking data set is constructed on monitoring videos shot by cameras in a plurality of public places, and labeling contents comprise a pedestrian bounding box id, a pedestrian id, a bounding box position, whether a pedestrian is shielded in the bounding box or not, and in which bounding box a pedestrian target is contained, wherein the information forms a truth value label in a training set and a testing set.
3. A multi-target tracking method based on coarse-to-fine occlusion processing as claimed in claim 1, wherein: and in the fourth step, the results in the first step and the second step are combined to obtain the positions and apparent feature vectors of all pedestrian targets in the current frame, and the characteristics of similarity and small position variation of the same apparent feature vector of the pedestrian target between adjacent frames are utilized to complete the matching of the pedestrian targets, so that the task is finally completed.
CN202111035065.6A 2021-09-05 2021-09-05 Multi-target tracking method based on coarse-to-fine shielding processing Active CN113763427B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111035065.6A CN113763427B (en) 2021-09-05 2021-09-05 Multi-target tracking method based on coarse-to-fine shielding processing

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111035065.6A CN113763427B (en) 2021-09-05 2021-09-05 Multi-target tracking method based on coarse-to-fine shielding processing

Publications (2)

Publication Number Publication Date
CN113763427A CN113763427A (en) 2021-12-07
CN113763427B true CN113763427B (en) 2024-02-23

Family

ID=78792988

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111035065.6A Active CN113763427B (en) 2021-09-05 2021-09-05 Multi-target tracking method based on coarse-to-fine shielding processing

Country Status (1)

Country Link
CN (1) CN113763427B (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114120373A (en) * 2022-01-24 2022-03-01 苏州浪潮智能科技有限公司 Model training method, device, equipment and storage medium
WO2023197232A1 (en) * 2022-04-14 2023-10-19 京东方科技集团股份有限公司 Target tracking method and apparatus, electronic device, and computer readable medium
CN116129432B (en) * 2023-04-12 2023-06-16 成都睿瞳科技有限责任公司 Multi-target tracking labeling method, system and storage medium based on image recognition

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112836639A (en) * 2021-02-03 2021-05-25 江南大学 Pedestrian multi-target tracking video identification method based on improved YOLOv3 model

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11055854B2 (en) * 2018-08-23 2021-07-06 Seoul National University R&Db Foundation Method and system for real-time target tracking based on deep learning

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112836639A (en) * 2021-02-03 2021-05-25 江南大学 Pedestrian multi-target tracking video identification method based on improved YOLOv3 model

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
基于YOLOv3与卡尔曼滤波的多目标跟踪算法;任珈民;宫宁生;韩镇阳;;计算机应用与软件(第05期);全文 *

Also Published As

Publication number Publication date
CN113763427A (en) 2021-12-07

Similar Documents

Publication Publication Date Title
CN113763427B (en) Multi-target tracking method based on coarse-to-fine shielding processing
EP1836683B1 (en) Method for tracking moving object in video acquired of scene with camera
Zhou et al. Efficient road detection and tracking for unmanned aerial vehicle
US20230289979A1 (en) A method for video moving object detection based on relative statistical characteristics of image pixels
She et al. Vehicle tracking using on-line fusion of color and shape features
CN101957997A (en) Regional average value kernel density estimation-based moving target detecting method in dynamic scene
Shukla et al. Moving object tracking of vehicle detection: a concise review
CN101916446A (en) Gray level target tracking algorithm based on marginal information and mean shift
CN105321189A (en) Complex environment target tracking method based on continuous adaptive mean shift multi-feature fusion
Tang et al. Multiple-kernel adaptive segmentation and tracking (MAST) for robust object tracking
Hu et al. A novel approach for crowd video monitoring of subway platforms
CN113706584A (en) Streetscape flow information acquisition method based on computer vision
Roy et al. A comprehensive survey on computer vision based approaches for moving object detection
CN116012949B (en) People flow statistics and identification method and system under complex scene
CN111986233B (en) Large-scene minimum target remote sensing video tracking method based on feature self-learning
Shbib et al. Distributed monitoring system based on weighted data fusing model
CN110322474B (en) Image moving target real-time detection method based on unmanned aerial vehicle platform
Davies et al. Using CART to segment road images
Streib et al. Extracting Pathlets FromWeak Tracking Data
Chanawangsa et al. A new color-based lane detection via Gaussian radial basis function networks
Bhuvaneshwar et al. Real-time detection of crossing pedestrians for traffic-adaptive signal control
Pulare et al. Implementation of real time multiple object detection and classification of HEVC videos
Chen et al. Pedestrian tracking algorithm based on Kalman filter and partial mean-shift tracking
Zhang et al. Research on single object tracking algorithm based on Siamese network and Kalman filter
Sanap et al. Survey on moving object detection

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant