CN109758756A - Gymnastics video analysis method and system based on 3D camera - Google Patents

Gymnastics video analysis method and system based on 3D camera Download PDF

Info

Publication number
CN109758756A
CN109758756A CN201910151559.7A CN201910151559A CN109758756A CN 109758756 A CN109758756 A CN 109758756A CN 201910151559 A CN201910151559 A CN 201910151559A CN 109758756 A CN109758756 A CN 109758756A
Authority
CN
China
Prior art keywords
camera
artis
gymnastics
crucial
human body
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201910151559.7A
Other languages
Chinese (zh)
Other versions
CN109758756B (en
Inventor
何卫
陆斌
苑廷刚
赵丹彤
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
CHINA INSTITUTE OF SPORTS SCIENCE
Original Assignee
CHINA INSTITUTE OF SPORTS SCIENCE
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by CHINA INSTITUTE OF SPORTS SCIENCE filed Critical CHINA INSTITUTE OF SPORTS SCIENCE
Priority to CN201910151559.7A priority Critical patent/CN109758756B/en
Publication of CN109758756A publication Critical patent/CN109758756A/en
Application granted granted Critical
Publication of CN109758756B publication Critical patent/CN109758756B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Abstract

The present invention relates to a kind of video analysis method and system for compensatory gymnastics training.The gymnastics video analysis method and analysis system that its purpose is to provide a kind of detection accuracy that can be improved human body key artis, realize data intelligent automatic processing.Gymnastics video analytic system the present invention is based on 3D camera includes 3D camera, analysis module and display module.Analysis method is to carry out pedestrian detection first, estimates human body key artis according to the candidate frame of pedestrian detection;Then with the crucial artis of convolutional neural networks prediction human body, to obtain accurate body joint point coordinate estimation, guarantees under different perspectives, illumination, distance change, relatively good result can be obtained;The colour information of image and depth information are combined again, obtain human body key artis 3D information;The auxiliary training system of gymnastics is finally formed in the track of 3d space and crucial artis angle information, building according to human body key artis.

Description

Gymnastics video analysis method and system based on 3D camera
Technical field
The present invention relates to a kind of auxiliary training systems, more particularly to a kind of video analysis system for compensatory gymnastics training System and method.
Background technique
In traditional sports training, trainer is generallyd use based on the training method for visually observing judgement.One Aspect, it is on the scene that this requires veteran practitioner such as to train, and on the other hand, this method cannot achieve automation, need to disappear A large amount of manpower and material resources are consumed to complete.
With the development of computer technology, sport training system systematic research is sent out towards intelligent and scientific direction Exhibition, more and more video processing techniques and image processing techniques are used in the analysis of sports.At present in gymnastics video Analysis field trainer mostly uses greatly two-dimensional video analytical technology to realize artis, angle, center of gravity and other correlations point Analysis.But these action process cannot be obtained since gymnastic has movement, the two-dimensional analysis technologies such as a large amount of overturning, rotation Valid data and information, cannot to scientific analysis provide data supporting, can not carry out multi-angle video observation, provide more Valuable visual information.
Visualization aspect is absorbed in the 3D analysis that gymnastics analysis field occurs at present with application mostly, only trains or transports Mobilize and the basic function intuitively analyzed of multi-angle be provided, be not bound with 3D analytical technology, depth learning technology current research at The space time information of key technology node in motion process is extracted in fruit, quantitative analysis, more without providing intelligence using the relevant technologies Change, the gymnastics electronic judgement function of autonomy-oriented.
A kind of existing training system example is a kind of auxiliary training system based on human body attitude algorithm for estimating, this method The coordinate that 15 major joint points of human body are extracted from monocular video builds the auxiliary instruction of golf on the basis of the above Practice system.Specific step is as follows for this method:
(1), target detection;Background modeling method based on ViBe model detects and extracts human body two-value profile in video Figure;
(2), contour edge feature extraction obtains the contour edge figure of image with Canny edge detection algorithm;
(3), human body attitude is estimated, obtains 15 body joint point coordinates of manikin in contour edge figure;
(4), auxiliary training system is built.
The disadvantages of this method is as follows:
(1), this method is cannot accurately to carry out the analysis of data based on two-dimensional analytic technique;
(2), this method is based on traditional digital image processing techniques, and the estimation of human joint points is inaccurate, uncomfortable For movements such as existing reversions a large amount of in gymnastic, rotations.
Summary of the invention
The technical problem to be solved in the present invention is to provide a kind of detection accuracy, realities that can be improved human body key artis The gymnastics video analytic system and analysis method of existing data intelligent automatic processing.
The present invention is based on the gymnastics video analysis methods of 3D camera, comprising the following steps:
A, captured in real-time is carried out to the gymnastic movement that sportsman makes using 3D camera, the video data after being synchronized;
B, video data is input in analysis module and obtains input data set, and input data set is pre-processed;
C, using convolutional neural networks, the candidate coordinate of each crucial artis of human body is obtained, the specific steps are that:
C1, the candidate frame of human body is obtained using pedestrian detection frame;
C2, the coordinate information that each crucial artis is obtained by several convolutional layers in convolutional neural networks;
D is obtained according to the inside and outside parameter of camera, the coordinate of crucial artis and each crucial corresponding depth information of artis The coordinate in 3d space of each key artis;
E obtains the angle information between artis and continuous track according to the 3D coordinate of obtained each crucial artis, To construct auxiliary training system.
The present invention is based on the gymnastics video analysis methods of 3D camera, wherein parameter synchronous in the step a be it is colored with Depth.
The present invention is based on the gymnastics video analysis methods of 3D camera, wherein pretreated content includes number in the step b According to extension and partitioned data set picture.
The present invention is based on the gymnastics video analysis methods of 3D camera, wherein using top-down (top- in the step c2 Down the coordinate information of each crucial artis of the acquisition of structure), be divided into two stages: the first stage uses faster- Rcnn principle detects, and detects multiple people in picture, and includes the rectangle frame of people to image according to each detected It is sheared;Second stage predicts thermal map (dense to the personage in each rectangle frame using the resnet network of full convolution Heatmap) and (offset) is deviated;The accurate positioning of key point is obtained finally by the fusion of thermal map and offset.
The present invention is based on the gymnastics video analytic system of 3D camera, including 3D camera, analysis module and display module, Analysis module includes input interface, processor and output interface, and the processor is built-in with convolutional neural networks model.
The present invention is based on the gymnastics video analytic systems of 3D camera, wherein the 3D camera passes through data line linking parsing mould The input interface of block, inside input interface, processor and the output interface of analysis module are successively electrically connected, output interface connection Display module.
The present invention is based on the gymnastics video analytic system differences from prior art of 3D camera to be that the present invention is based on 3D The gymnastics video analytic system and method for camera can obtain the data information of gymnastic key node by cordless, The data such as related angle, center of gravity, coordinate are further analyzed, is intuitively shown by visualization, is mentioned for the judge of coach For real-time, online data supporting, to solve inaccurate data in the prior art, data processing and analysis not smart enoughization Problem.
Specific embodiment
Gymnastics video analytic system the present invention is based on 3D camera is the training optimization of the neural network model based on depth Come what is completed, including 3D camera, analysis module and display module, analysis module include input interface, processor and output interface. 3D camera passes through the input interface of data line linking parsing module, and inside input interface, processor and the output of analysis module connect Mouth is successively electrically connected, and output interface connects display module.Wherein 3D camera is for acquiring video data, and analysis module is for locating Manage the data that 3D camera uses, display module treated for showing analysis module final output.
It is a kind of application the present invention is based on the analysis methods of the gymnastics video analytic system of 3D camera to mainly comprise the steps that
A: captured in real-time is carried out to the gymnastic movement that sportsman makes using 3D camera, is obtained colored synchronous with depth parameter Video data afterwards;
B: video data being input in analysis module and obtains input data set, and is pre-processed to input data set, Including Data expansion and partitioned data set picture;
C: utilizing convolutional neural networks, obtains the candidate coordinate of each crucial artis of human body, specific steps are as follows:
C1: obtaining the candidate frame of human body using pedestrian detection frame,
C2: the coordinate information of each crucial artis is obtained by several convolutional layers in convolutional neural networks;
D: each crucial joint is obtained according to the inside and outside parameter of camera, the coordinate of crucial artis and corresponding depth information The coordinate in 3d space of point;
E: according to the 3D coordinate of obtained each crucial artis, the angle information and continuous rail between artis are obtained Mark, to construct auxiliary training system.
In step c2, the coordinate using each crucial artis of the acquisition of the structure of top-down (top-down) is believed Breath, be divided into two stages: the first stage is detected using faster-rcnn principle, detects multiple people in picture, and according to Each detected includes that the rectangle frame of people shears image;Second stage is using the resnet network of full convolution to every Personage in one rectangle frame predicts thermal map (dense heatmap) and offset (offset);Finally by thermal map and offset Fusion obtains the accurate positioning of key point.
In step d, obtaining 3D coordinate can be according to formula:
It obtains.The process of camera imaging is exactly the process that world coordinate system is converted to pixel coordinate system, it may be assumed that world coordinate system (3d) → camera coordinates system (3d) → photo coordinate system (2d) → pixel coordinate system (2d), by conversion so step by step it Afterwards, the 3D coordinate of object in space is the pixel coordinate be converted in the picture.Specifically includes the following steps:
(1) world coordinate system (3d) -> camera coordinates system (3d):
Conversion from world coordinate system to camera coordinates system is rigid body translation, be spinning movement and translation motion as a result, Shown in following formula:
(2) camera coordinates system (3d) -> photo coordinate system (2d), shown in following formula:
(3) photo coordinate system (2d) -> pixel coordinate system (2d):
Pixel coordinate is an analog quantity of the light in planar imaging, thus need on imaging plane picture carry out sampling and Quantization obtains coordinate value of the picture of object on pixel planes.Between pixel planes and imaging plane, a scaling and original are differed The translation of point.It is shown below, is exaggerated α times on u axis, amplify β times on v axis, origin translation cx, cy.Obtain formula:
Here the influence of camera distortion is had ignored.
Fusion type 2), 3) obtain camera coordinates system to pixel coordinate conversion formula:
It is organized into homogeneous form:
It merges 5) and 1) obtains:
Wherein xw、yw、zwFor world coordinate system coordinate, xc、yc、zcFor camera coordinates system coordinate, x, y are photo coordinate system Coordinate, u, v are pixel coordinate system coordinate, and f is the distance between imaging plane and optical center.
Gymnastics video analysis method based on 3D camera of the invention carries out pedestrian detection first in implementation process, according to The candidate frame of pedestrian detection estimates human body key artis;The crucial artis of human body is then predicted with convolutional neural networks, To obtain accurate body joint point coordinate estimation, guarantees under different perspectives, illumination, distance change, can obtain relatively good Result;The colour information of image and depth information are combined again, obtain human body key artis 3D information;Finally according to human body Crucial artis forms the auxiliary training system of gymnastics in the track of 3d space and crucial artis angle information, building.
Gymnastics video analysis method based on 3D camera of the invention can not only provide the 3 D video image of multi-angle of view, Crucial artis and critical angle as this kind of body overturning of gymnastics, torsion action can also be extracted.The present invention is deep using convolution Learning neural network structure is spent, so that the accuracy of the detection of human body key artis significantly improves, to realize intelligence certainly The gymnastics auxiliary training system of dynamicization.
The beneficial effects of the present invention are the data information of gymnastic key node can be obtained by cordless, The data such as related angle, center of gravity, coordinate are further analyzed, is intuitively shown by visualization, is mentioned for the judge of coach For real-time, online data supporting, to solve the problems, such as that data are inaccurate in the prior art, data processing not smart enoughization.
Embodiment described above only describe the preferred embodiments of the invention, not to model of the invention It encloses and is defined, without departing from the spirit of the design of the present invention, those of ordinary skill in the art are to technical side of the invention The various changes and improvements that case is made should all be fallen into the protection scope that claims of the present invention determines.

Claims (6)

1. a kind of gymnastics video analysis method based on 3D camera, it is characterised in that: the following steps are included:
A, captured in real-time is carried out to the gymnastic movement that sportsman makes using 3D camera, the video data after being synchronized;
B, video data is input in analysis module and obtains input data set, and input data set is pre-processed;
C, using convolutional neural networks, the candidate coordinate of each crucial artis of human body is obtained, the specific steps are that:
C1, the candidate frame of human body is obtained using pedestrian detection frame;
C2, the coordinate information that each crucial artis is obtained by several convolutional layers in convolutional neural networks;
D, it is obtained according to the inside and outside parameter of camera, the coordinate of crucial artis and each crucial corresponding depth information of artis each The coordinate in 3d space of crucial artis;
E, according to the 3D coordinate of obtained each crucial artis, the angle information between artis and continuous track are obtained, from And construct auxiliary training system.
2. the gymnastics video analysis method according to claim 1 based on 3D camera, it is characterised in that: in the step a Synchronous parameter is colored and depth.
3. the gymnastics video analysis method according to claim 1 based on 3D camera, it is characterised in that: in the step b Pretreated content includes Data expansion and partitioned data set picture.
4. the gymnastics video analysis method according to claim 1 based on 3D camera, it is characterised in that: in the step c2 Using the coordinate information of each crucial artis of the acquisition of the structure of top-down (top-down), it is divided into two stages: first Stage is detected using faster-rcnn principle, detects multiple people in picture, and includes people according to each detected Rectangle frame image is sheared;Second stage is pre- to the personage in each rectangle frame using the resnet network of full convolution Calorimetric figure (dense heatmap) and offset (offset);The accurate of key point is obtained finally by the fusion of thermal map and offset Positioning.
5. a kind of gymnastics video analytic system based on 3D camera, it is characterised in that: including 3D camera, analysis module and display mould Block, analysis module include input interface, processor and output interface, and the processor is built-in with convolutional neural networks model.
6. the gymnastics video analytic system according to claim 5 based on 3D camera, it is characterised in that: the 3D camera is logical Cross the input interface of data line linking parsing module, the inside input interface of analysis module, processor and output interface are successively electric Property connection, output interface connect display module.
CN201910151559.7A 2019-02-28 2019-02-28 Gymnastics video analysis method and system based on 3D camera Active CN109758756B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910151559.7A CN109758756B (en) 2019-02-28 2019-02-28 Gymnastics video analysis method and system based on 3D camera

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910151559.7A CN109758756B (en) 2019-02-28 2019-02-28 Gymnastics video analysis method and system based on 3D camera

Publications (2)

Publication Number Publication Date
CN109758756A true CN109758756A (en) 2019-05-17
CN109758756B CN109758756B (en) 2021-03-23

Family

ID=66457443

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910151559.7A Active CN109758756B (en) 2019-02-28 2019-02-28 Gymnastics video analysis method and system based on 3D camera

Country Status (1)

Country Link
CN (1) CN109758756B (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110354481A (en) * 2019-07-30 2019-10-22 天水师范学院 A kind of athletic training analysis system based on digital field and high-speed image
CN110688929A (en) * 2019-09-20 2020-01-14 北京华捷艾米科技有限公司 Human skeleton joint point positioning method and device
CN111260649A (en) * 2020-05-07 2020-06-09 常州唯实智能物联创新中心有限公司 Close-range mechanical arm sensing and calibrating method
CN112102947A (en) * 2020-04-13 2020-12-18 国家体育总局体育科学研究所 Apparatus and method for body posture assessment
CN112668549A (en) * 2021-01-15 2021-04-16 北京格灵深瞳信息技术股份有限公司 Pedestrian attitude analysis method, system, terminal and storage medium
CN113379930A (en) * 2021-05-25 2021-09-10 广州紫为云科技有限公司 Immersive interaction method and device through human body graph and storage medium

Citations (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102072706A (en) * 2009-11-20 2011-05-25 深圳先进技术研究院 Multi-camera positioning and tracking method and system
US20170308856A1 (en) * 2007-09-28 2017-10-26 Great-Circle Technologies, Inc. Bundling of automated work flow
CN108335350A (en) * 2018-02-06 2018-07-27 聊城大学 The three-dimensional rebuilding method of binocular stereo vision
CN108416321A (en) * 2018-03-23 2018-08-17 北京市商汤科技开发有限公司 For predicting that target object moves method, control method for vehicle and the device of direction
CN108549876A (en) * 2018-04-20 2018-09-18 重庆邮电大学 The sitting posture detecting method estimated based on target detection and human body attitude
CN108805064A (en) * 2018-05-31 2018-11-13 中国农业大学 A kind of fish detection and localization and recognition methods and system based on deep learning
CN108829232A (en) * 2018-04-26 2018-11-16 深圳市深晓科技有限公司 The acquisition methods of skeleton artis three-dimensional coordinate based on deep learning
CN108875588A (en) * 2018-05-25 2018-11-23 武汉大学 Across camera pedestrian detection tracking based on deep learning
CN108898191A (en) * 2018-09-26 2018-11-27 苏州米特希赛尔人工智能有限公司 Convolutional neural networks feature-extraction images sensor
US20180341495A1 (en) * 2017-05-26 2018-11-29 Purdue Research Foundation Hardware Accelerator for Convolutional Neural Networks and Method of Operation Thereof
CN208172859U (en) * 2018-04-28 2018-11-30 上海视可电子科技有限公司 A kind of intelligent image acquisition device
CN108972494A (en) * 2018-06-22 2018-12-11 华南理工大学 A kind of Apery manipulator crawl control system and its data processing method
CN109087329A (en) * 2018-07-27 2018-12-25 中山大学 Human body three-dimensional joint point estimation frame and its localization method based on depth network
CN109145788A (en) * 2018-08-08 2019-01-04 北京云舶在线科技有限公司 Attitude data method for catching and system based on video
CN109241841A (en) * 2018-08-01 2019-01-18 甘肃未来云数据科技有限公司 The acquisition methods and device of video human movement
CN109359636A (en) * 2018-12-14 2019-02-19 腾讯科技(深圳)有限公司 Video classification methods, device and server
CN109376681A (en) * 2018-11-06 2019-02-22 广东工业大学 A kind of more people's Attitude estimation method and system

Patent Citations (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170308856A1 (en) * 2007-09-28 2017-10-26 Great-Circle Technologies, Inc. Bundling of automated work flow
CN102072706A (en) * 2009-11-20 2011-05-25 深圳先进技术研究院 Multi-camera positioning and tracking method and system
US20180341495A1 (en) * 2017-05-26 2018-11-29 Purdue Research Foundation Hardware Accelerator for Convolutional Neural Networks and Method of Operation Thereof
CN108335350A (en) * 2018-02-06 2018-07-27 聊城大学 The three-dimensional rebuilding method of binocular stereo vision
CN108416321A (en) * 2018-03-23 2018-08-17 北京市商汤科技开发有限公司 For predicting that target object moves method, control method for vehicle and the device of direction
CN108549876A (en) * 2018-04-20 2018-09-18 重庆邮电大学 The sitting posture detecting method estimated based on target detection and human body attitude
CN108829232A (en) * 2018-04-26 2018-11-16 深圳市深晓科技有限公司 The acquisition methods of skeleton artis three-dimensional coordinate based on deep learning
CN208172859U (en) * 2018-04-28 2018-11-30 上海视可电子科技有限公司 A kind of intelligent image acquisition device
CN108875588A (en) * 2018-05-25 2018-11-23 武汉大学 Across camera pedestrian detection tracking based on deep learning
CN108805064A (en) * 2018-05-31 2018-11-13 中国农业大学 A kind of fish detection and localization and recognition methods and system based on deep learning
CN108972494A (en) * 2018-06-22 2018-12-11 华南理工大学 A kind of Apery manipulator crawl control system and its data processing method
CN109087329A (en) * 2018-07-27 2018-12-25 中山大学 Human body three-dimensional joint point estimation frame and its localization method based on depth network
CN109241841A (en) * 2018-08-01 2019-01-18 甘肃未来云数据科技有限公司 The acquisition methods and device of video human movement
CN109145788A (en) * 2018-08-08 2019-01-04 北京云舶在线科技有限公司 Attitude data method for catching and system based on video
CN108898191A (en) * 2018-09-26 2018-11-27 苏州米特希赛尔人工智能有限公司 Convolutional neural networks feature-extraction images sensor
CN109376681A (en) * 2018-11-06 2019-02-22 广东工业大学 A kind of more people's Attitude estimation method and system
CN109359636A (en) * 2018-12-14 2019-02-19 腾讯科技(深圳)有限公司 Video classification methods, device and server

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110354481A (en) * 2019-07-30 2019-10-22 天水师范学院 A kind of athletic training analysis system based on digital field and high-speed image
CN110688929A (en) * 2019-09-20 2020-01-14 北京华捷艾米科技有限公司 Human skeleton joint point positioning method and device
CN110688929B (en) * 2019-09-20 2021-11-30 北京华捷艾米科技有限公司 Human skeleton joint point positioning method and device
CN112102947A (en) * 2020-04-13 2020-12-18 国家体育总局体育科学研究所 Apparatus and method for body posture assessment
CN112102947B (en) * 2020-04-13 2024-02-13 国家体育总局体育科学研究所 Apparatus and method for body posture assessment
CN111260649A (en) * 2020-05-07 2020-06-09 常州唯实智能物联创新中心有限公司 Close-range mechanical arm sensing and calibrating method
CN112668549A (en) * 2021-01-15 2021-04-16 北京格灵深瞳信息技术股份有限公司 Pedestrian attitude analysis method, system, terminal and storage medium
CN113379930A (en) * 2021-05-25 2021-09-10 广州紫为云科技有限公司 Immersive interaction method and device through human body graph and storage medium

Also Published As

Publication number Publication date
CN109758756B (en) 2021-03-23

Similar Documents

Publication Publication Date Title
CN109758756A (en) Gymnastics video analysis method and system based on 3D camera
CN108388882B (en) Gesture recognition method based on global-local RGB-D multi-mode
CN104167016B (en) A kind of three-dimensional motion method for reconstructing based on RGB color and depth image
Hoppe et al. Online Feedback for Structure-from-Motion Image Acquisition.
CN108898063A (en) A kind of human body attitude identification device and method based on full convolutional neural networks
CN108256439A (en) A kind of pedestrian image generation method and system based on cycle production confrontation network
CN101520902A (en) System and method for low cost motion capture and demonstration
CN112990310A (en) Artificial intelligence system and method for serving electric power robot
CN110310306A (en) Method for tracking target, system and medium based on outdoor scene modeling and intelligent recognition
CN106780543A (en) A kind of double framework estimating depths and movement technique based on convolutional neural networks
CN111680586B (en) Badminton player motion attitude estimation method and system
CN106156714A (en) The Human bodys' response method merged based on skeletal joint feature and surface character
CN112101262B (en) Multi-feature fusion sign language recognition method and network model
CN110941996A (en) Target and track augmented reality method and system based on generation of countermeasure network
CN106815855A (en) Based on the human body motion tracking method that production and discriminate combine
CN110009675A (en) Generate method, apparatus, medium and the equipment of disparity map
CN111145865A (en) Vision-based hand fine motion training guidance system and method
CN115376034A (en) Motion video acquisition and editing method and device based on human body three-dimensional posture space-time correlation action recognition
CN112365578A (en) Three-dimensional human body model reconstruction system and method based on double cameras
CN113065506A (en) Human body posture recognition method and system
CN106964137B (en) A kind of ball service behavior rating method based on image procossing
CN112288876A (en) Long-distance AR identification server and system
CN111461772A (en) Video advertisement integration system and method based on generation countermeasure network
CN113920254B (en) Monocular RGB (Red Green blue) -based indoor three-dimensional reconstruction method and system thereof
CN116030533A (en) High-speed motion capturing and identifying method and system for motion scene

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant