CN114187219A - Moving target shadow real-time elimination method based on red, green and blue double difference - Google Patents

Moving target shadow real-time elimination method based on red, green and blue double difference Download PDF

Info

Publication number
CN114187219A
CN114187219A CN202111476285.2A CN202111476285A CN114187219A CN 114187219 A CN114187219 A CN 114187219A CN 202111476285 A CN202111476285 A CN 202111476285A CN 114187219 A CN114187219 A CN 114187219A
Authority
CN
China
Prior art keywords
pixel point
green
red
blue
difference
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202111476285.2A
Other languages
Chinese (zh)
Other versions
CN114187219B (en
Inventor
刘景贤
刘盛
何梦龙
杨锦铸
梁杰
沈翱翔
陈德文
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangxi University of Science and Technology
Original Assignee
Guangxi University of Science and Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangxi University of Science and Technology filed Critical Guangxi University of Science and Technology
Priority to CN202111476285.2A priority Critical patent/CN114187219B/en
Publication of CN114187219A publication Critical patent/CN114187219A/en
Application granted granted Critical
Publication of CN114187219B publication Critical patent/CN114187219B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/50Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/20Analysis of motion
    • G06T7/254Analysis of motion involving subtraction of images
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10024Color image
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20212Image combination
    • G06T2207/20224Image subtraction

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Analysis (AREA)

Abstract

The invention aims to provide a method for eliminating a shadow of a moving target in real time based on red, green and blue double differences, which comprises the following steps: A. shooting a background frame RGB image, and shooting a current frame RGB image containing a target; B. respectively subtracting each pixel point in the current frame RGB image from the corresponding pixel point in the background frame RGB image according to red, green and blue three channels; C. mutually subtracting the one-time difference results of the red, green and blue three-channel pixel points of each pixel point again; D. performing mean calculation on the double difference result of each pixel point according to the corresponding point to obtain the mean value of each pixel point; E. setting a threshold, comparing each pixel point with the threshold, and setting the pixel point value to be 255 if the pixel point value is larger than the threshold; and if the pixel point value is less than or equal to the threshold value, setting the pixel point value to be 0, and constructing a single-channel binary image. The method can effectively eliminate the influence of the shadow on the target and detect the pixel points of the target.

Description

Moving target shadow real-time elimination method based on red, green and blue double difference
Technical Field
The invention relates to the field of image processing, in particular to a moving target shadow real-time elimination method based on red, green and blue double differences.
Background
The shadow is a natural phenomenon generated by the fact that a light source is shielded by an object, and the target detection and tracking effects are seriously influenced. The main methods for eliminating the shadow include a model method and a feature method.
Modeling methods typically design models based on known shadow prior knowledge and detect and eliminate shadows accordingly. For example: zhang et al propose a robust vehicle detection method with shadow elimination, but the method for removing shadow regions by using a model proposed by the method is only suitable for relevant regions mentioned in the article, and the application range is small. Benish et al propose a technique for segmenting shadow regions using a tristimulus model (TAM) and intensity information, but this technique still requires the manual selection of shadow regions when removing shadows. Murali et al propose a technique for removing shadows from an image with uniform texture using modeling of shadow regions, but the technique is still highly dependent on the effect of the modeling. Since the shadow of the moving object itself changes as it moves, determining an effective prior is very difficult and limiting. Therefore, the shadow elimination effect of the model method on the moving target is not ideal.
The eigen method focuses on detecting and eliminating shadows using similar changes in the physical characteristics of the shadow region itself, which involve physical characteristics including brightness, color, texture, gradients, edges, etc. Generally, the physical characteristics of shadows are relatively stable, so that the characteristic method is most widely applied to the detection of the current moving objects. For example, the image log domain difference thought is utilized by the allowed macroscopics and the like to weaken the shadow area firstly and then eliminate the shadow through HSV color characteristics; the Park is to remove the shadow by using a shadow depth map and an illumination invariant feature; liyawei and the like provide a shadow weakening algorithm based on brightness and texture characteristics, and do not need early training and manual intervention; chen et al propose to detect shadows by RGB color and further detect and eliminate shadow pixels based on the distribution rule of vector distance normalized NVD. However, the physical quantities such as the brightness and the color of the image are greatly affected by the illumination and the camera imaging method, so that the stability is not high. The physical quantities such as texture, gradient, edge, etc. are highly dependent on the hyper-parameters involved in the algorithm, and thus the applicability is less than ideal. Although the latest shadow elimination technology at present sufficiently fuses shadow detection results of different physical characteristics to improve the stability of shadow elimination, the effect is still not ideal enough, and a large calculation overhead is additionally added, so that the requirements of real-time detection and elimination of the target shadow are difficult to meet.
Disclosure of Invention
The invention aims to provide a moving target shadow real-time elimination method based on red-green-blue double difference, which can effectively eliminate the influence of shadow on a target and correctly detect pixel points of the target.
The technical scheme of the invention is as follows:
the method for eliminating the shadow of the moving target in real time based on the red, green and blue double difference comprises the following steps:
A. photographing a background frame RGB image Bm×n×3Shooting a current frame RGB image F containing the targetm×n×3
B. The current frame RGB image Fm×n×3Each pixel point in the image and the corresponding pixel point in the background frame RGB image Bm×n×3The corresponding pixel points in the middle are respectively subjected to difference according to the red, green and blue channels to obtain a double difference result delta R of the pixel points of the red channelm×nOne-time difference result delta G of green channel pixel pointsm×nOne-time difference result delta B of blue channel pixel pointsm×n
C. One-time difference result Delta R of each pixel pointm×n、△Gm×n、△Bm×nMaking difference again to obtain the double difference result delta of red and green channels of each pixel point respectively2RGm×nDouble difference result of red and blue channels2RBm×nGreen-blue channel double difference result delta2GBm×n
D. The double difference result delta of each pixel point2RGm×n,△2RBm×n,△2GBm×nCalculating the mean value according to the corresponding points to obtain the mean value delta of each pixel point2AVGm×n
E. Setting threshold value, delta of each pixel point2AVGm×nRespectively comparing with a threshold value, and setting the pixel point value to be 255 if the pixel point value is larger than the threshold value; and if the pixel point value is less than or equal to the threshold value, setting the pixel point value to be 0, constructing a single-channel binary image, and obtaining the target current frame image with the shadow eliminated.
In the step B, the formula for respectively making difference of the red, green and blue three channels is as follows:
[△Rm×n,△Gm×n,△Bm×n]=△Fm×n×3=Bm×n×3-Fm×n×3 (1)。
in the step C, the formula of mutually subtracting the multiple difference results of the red, green and blue three-channel pixel points of the pixel point is as follows:
2RGm×n=△Rm×n-△Gm×n (2)
2RBm×n=△Rm×n-△Bm×n (3)
2GBm×n=△Gm×n-△Bm×n (4)。
in the step D, the mean value calculation formula is as follows:
Figure BDA0003393660500000021
in the above step E, the threshold value is set to 30.
The invention adopts the thought of double difference to calculate the relationship between the current frame and the background frame of the image, utilizes the characteristic that the difference values of the red, green and blue three-channel pixel points in the target shadow of the color image are similar to the difference values of the original background, respectively differentiates the three-channel pixel point difference values, then calculates the mean value of the corresponding points of the three double differences through double difference, ensures the stability of the shadow pixel points in the double difference calculation, eliminates the shadow area through a threshold value, and directly extracts the binary image of the target area.
Drawings
Fig. 1 is a background frame RGB image of embodiment 1;
FIG. 2 is an image of a target at the time of example 1, which is disturbed by shadows;
FIG. 3 is an image of a target at time two in example 1, which is disturbed by shadows;
FIG. 4 is an image of the three targets affected by shadow at time instant of example 1;
FIG. 5 is a binary diagram of the instant-target with shadow interference removed in example 1;
FIG. 6 is a binary diagram of the two targets at time instant 1 after eliminating shadow interference;
FIG. 7 is a binary diagram of the three targets eliminating the shadow interference at time instant of embodiment 1;
Detailed Description
The present invention will be described in detail with reference to the accompanying drawings and examples.
Example 1
The method for eliminating the shadow of the moving target in real time based on the red, green and blue double difference comprises the following steps:
A. photographing a background frame RGB image Bm×n×3Shooting a current frame RGB image F containing the targetm×n×3
B. The current frame RGB image Fm×n×3Each pixel point in the image and the corresponding pixel point in the background frame RGB image Bm×n×3The corresponding pixel points in the middle are respectively subjected to difference according to the red, green and blue channels to obtain a double difference result delta R of the pixel points of the red channelm×nOne-time difference result delta G of green channel pixel pointsm×nOne-time difference result delta B of blue channel pixel pointsm×n
The formula for making difference of red, green and blue channels is:
[△Rm×n,△Gm×n,△Bm×n]=△Fm×n×3=Bm×n×3-Fm×n×3 (1);
C. one-time difference result Delta R of each pixel pointm×n、△Gm×n、△Bm×nMutually make difference again to respectively obtainRed and green channel double difference result delta to each pixel point2RGm×nDouble difference result of red and blue channels2RBm×nGreen-blue channel double difference result delta2GBm×n
The formula for mutually subtracting the one-time difference results of the red, green and blue three-channel pixel points of the pixel point is as follows:
2RGm×n=△Rm×n-△Gm×n (2)
2RBm×n=△Rm×n-△Bm×n (3)
2GBm×n=△Gm×n-△Bm×n (4);
D. the double difference result delta of each pixel point2RGm×n,△2RBm×n,△2GBm×nCalculating the mean value according to the corresponding points to obtain the mean value delta of each pixel point2AVGm×n
Figure BDA0003393660500000041
E. Setting the threshold value to be 30 and delta of each pixel point2AVGm×nRespectively comparing with a threshold value, and setting the pixel point value to be 255 if the pixel point value is larger than the threshold value; and if the pixel point value is less than or equal to the threshold value, setting the pixel point value to be 0, constructing a single-channel binary image, and obtaining the target current frame image with the shadow eliminated.
The experimental results are shown in fig. 1-7, where fig. 1 is a background frame RGB image, fig. 2 is a time-one-target shadow interference image, fig. 3 is a time-two-target shadow interference image, fig. 4 is a time-three-target shadow interference image, fig. 5 is a time-one-target binary image after shadow interference is eliminated, fig. 6 is a time-two-target binary image after shadow interference is eliminated, and fig. 7 is a time-three-target binary image after shadow interference is eliminated.
The background frame and the current frame at three randomly selected moments are used as input, and the binary image of the current frame after shadow elimination can be finally obtained through calculation of the design. Fig. 5-7 represent the target binary frame data after the target is removed from the shadow interference in fig. 2-4, respectively. Obviously, the method can effectively eliminate the influence of the shadow on the target and correctly detect the pixel points of the target.

Claims (5)

1.一种基于红绿蓝二重差分的移动目标阴影实时消除方法,其特征在于包括以下步骤:1. a moving target shadow real-time elimination method based on red, green and blue double difference is characterized in that comprising the following steps: A、拍摄背景帧RGB图像Bm×n×3,拍摄包含目标的当前帧RGB图像Fm×n×3A. Shoot the background frame RGB image B m×n×3 , and shoot the current frame RGB image F m×n×3 including the target; B、将当前帧RGB图像Fm×n×3中的各个像素点分别与其在背景帧RGB图像Bm×n×3中对应的像素点,按照红、绿、蓝三通道分别做差,得到红色通道像素点的一重差分结果△Rm×n、绿色通道像素点的一重差分结果△Gm×n、蓝色通道像素点的一重差分结果△Bm×nB. Differentiate each pixel point in the current frame RGB image F m×n×3 and its corresponding pixel point in the background frame RGB image B m×n×3 respectively according to the red, green and blue channels to obtain The single difference result ΔR m×n of the red channel pixel point, the single difference result ΔG m×n of the green channel pixel point, and the single difference result ΔB m ×n of the blue channel pixel point; C、将各个像素点的一重差分结果△Rm×n、△Gm×n、△Bm×n相互再次做差,分别得到各个像素点的红绿通道二重差分结果△2RGm×n、红蓝通道二重差分结果△2RBm×n、绿蓝通道二重差分结果△2GBm×nC. Make the difference between the single difference results of each pixel point △R m×n , △G m×n , △B m×n again, and obtain the red and green channel double difference result of each pixel point △ 2 RG m× n , the red and blue channel double difference result △ 2 RB m×n , the green and blue channel double difference result △ 2 GB m×n ; D、将各个像素点的二重差分结果△2RGm×n,△2RBm×n,△2GBm×n按对应点做均值计算,得到各个像素点的均值△2AVGm×nD. Calculate the average value of the corresponding points according to the double difference results of each pixel △ 2 RG m×n , △ 2 RB m×n , △ 2 GB m×n , and obtain the average value of each pixel point △ 2 AVG m×n ; E、设定阈值,各个像素点的△2AVGm×n分别与阈值比较,大于阈值的,像素点值设定为255;小于等于阈值的,像素点值设定为0,构建出单通道二值图像,得到阴影消除后的目标当前帧图像。E. Set the threshold. The △ 2 AVG m×n of each pixel is compared with the threshold. If it is greater than the threshold, the pixel value is set to 255; if it is less than or equal to the threshold, the pixel value is set to 0, and a single channel is constructed. A binary image is obtained to obtain the target current frame image after the shadow is eliminated. 2.如权利要求1所述的基于红绿蓝二重差分的移动目标阴影实时消除方法,其特征在于:2. the moving target shadow real-time elimination method based on red, green and blue double difference as claimed in claim 1, is characterized in that: 所述的步骤B中,红、绿、蓝三通道分别做差的公式为:In the described step B, the formulas for making the difference between the red, green and blue channels respectively are: [△Rm×n,△Gm×n,△Bm×n]=△Fm×n×3=Bm×n×3-Fm×n×3 (1)。[ΔR m×n , ΔG m×n , ΔB m×n ]=ΔF m×n×3 =B m×n×3 −F m×n×3 (1). 3.如权利要求1所述的基于红绿蓝二重差分的移动目标阴影实时消除方法,其特征在于:3. the moving target shadow real-time elimination method based on red, green and blue double difference as claimed in claim 1, is characterized in that: 所述的步骤C中,像素点的红、绿、蓝三通道像素点的一重差分结果相互再次做差的公式为:In the described step C, the formula for making the difference between the red, green, and blue three-channel pixel points of the pixel point by one difference is as follows: 2RGm×n=△Rm×n-△Gm×n (2)2 RG m×n =△R m×n -△G m×n (2) 2RBm×n=△Rm×n-△Bm×n (3)2 RB m×n =△R m×n -△B m×n (3) 2GBm×n=△Gm×n-△Bm×n (4)。Δ 2 GB m×n =ΔG m×n −ΔB m×n (4). 4.如权利要求1所述的基于红绿蓝二重差分的移动目标阴影实时消除方法,其特征在于:4. the moving target shadow real-time elimination method based on red, green and blue double difference as claimed in claim 1, is characterized in that: 所述的步骤D中,均值计算公式为:In the described step D, the mean value calculation formula is:
Figure FDA0003393660490000021
Figure FDA0003393660490000021
5.如权利要求1所述的基于红绿蓝二重差分的移动目标阴影实时消除方法,其特征在于:5. the moving target shadow real-time elimination method based on red, green and blue double difference as claimed in claim 1, is characterized in that: 所述的步骤E中,阈值设定为30。In the step E, the threshold is set to 30.
CN202111476285.2A 2021-12-06 2021-12-06 Moving target shadow real-time eliminating method based on red-green-blue double difference Active CN114187219B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111476285.2A CN114187219B (en) 2021-12-06 2021-12-06 Moving target shadow real-time eliminating method based on red-green-blue double difference

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111476285.2A CN114187219B (en) 2021-12-06 2021-12-06 Moving target shadow real-time eliminating method based on red-green-blue double difference

Publications (2)

Publication Number Publication Date
CN114187219A true CN114187219A (en) 2022-03-15
CN114187219B CN114187219B (en) 2024-06-25

Family

ID=80603458

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111476285.2A Active CN114187219B (en) 2021-12-06 2021-12-06 Moving target shadow real-time eliminating method based on red-green-blue double difference

Country Status (1)

Country Link
CN (1) CN114187219B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115482474A (en) * 2022-08-24 2022-12-16 湖南科技大学 A method and system for identifying vehicle loads on bridge decks based on high-altitude aerial images

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090087025A1 (en) * 2007-09-29 2009-04-02 Samsung Electronics Co., Ltd. Shadow and highlight detection system and method of the same in surveillance camera and recording medium thereof
CN101833749A (en) * 2009-03-12 2010-09-15 株式会社理光 Device and method for detecting shadow in image
KR20100117976A (en) * 2009-04-27 2010-11-04 삼성테크윈 주식회사 Apparatus for estimation shadow
CN102147861A (en) * 2011-05-17 2011-08-10 北京邮电大学 Moving target detection method for carrying out Bayes judgment based on color-texture dual characteristic vectors
CN103164847A (en) * 2013-04-03 2013-06-19 上海理工大学 Method for eliminating shadow of moving target in video image
CN103295013A (en) * 2013-05-13 2013-09-11 天津大学 Pared area based single-image shadow detection method
CN103440628A (en) * 2013-08-27 2013-12-11 宁波海视智能系统有限公司 Method for removing shadow interference of moving object in video
CN105913441A (en) * 2016-04-27 2016-08-31 四川大学 Shadow removal method for improving target detection performance in video
CN107767390A (en) * 2017-10-20 2018-03-06 苏州科达科技股份有限公司 The shadow detection method and its system of monitor video image, shadow removal method
CN112949484A (en) * 2021-03-01 2021-06-11 浙江大学 High-speed railway rockfall real-time detection method based on aggregated channel features and texture features

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090087025A1 (en) * 2007-09-29 2009-04-02 Samsung Electronics Co., Ltd. Shadow and highlight detection system and method of the same in surveillance camera and recording medium thereof
CN101833749A (en) * 2009-03-12 2010-09-15 株式会社理光 Device and method for detecting shadow in image
KR20100117976A (en) * 2009-04-27 2010-11-04 삼성테크윈 주식회사 Apparatus for estimation shadow
CN102147861A (en) * 2011-05-17 2011-08-10 北京邮电大学 Moving target detection method for carrying out Bayes judgment based on color-texture dual characteristic vectors
CN103164847A (en) * 2013-04-03 2013-06-19 上海理工大学 Method for eliminating shadow of moving target in video image
CN103295013A (en) * 2013-05-13 2013-09-11 天津大学 Pared area based single-image shadow detection method
CN103440628A (en) * 2013-08-27 2013-12-11 宁波海视智能系统有限公司 Method for removing shadow interference of moving object in video
CN105913441A (en) * 2016-04-27 2016-08-31 四川大学 Shadow removal method for improving target detection performance in video
CN107767390A (en) * 2017-10-20 2018-03-06 苏州科达科技股份有限公司 The shadow detection method and its system of monitor video image, shadow removal method
CN112949484A (en) * 2021-03-01 2021-06-11 浙江大学 High-speed railway rockfall real-time detection method based on aggregated channel features and texture features

Non-Patent Citations (5)

* Cited by examiner, † Cited by third party
Title
李萌;郑娟毅;门瑜;: "基于帧差法的运动车辆阴影去除算法", 电视技术, no. 10, 17 October 2016 (2016-10-17) *
王静静;林明秀;魏颖;: "基于灰度相关的帧间差分和背景差分相融合的实时目标检测", 中南大学学报(自然科学版), no. 1, 10 September 2009 (2009-09-10) *
管业鹏;顾伟康;: "二维场景阴影区域的自动鲁棒分割", 电子学报, no. 04, 25 April 2006 (2006-04-25) *
胡金杰;王远弟;: "视频监控系统中运动目标的阴影消除", 应用数学与计算数学学报, no. 01, 15 June 2010 (2010-06-15) *
陈前: "复杂背景下运动目标检测和阴影消除算法研究", 《中国优秀硕士学位论文全文数据库》, 15 February 2020 (2020-02-15) *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115482474A (en) * 2022-08-24 2022-12-16 湖南科技大学 A method and system for identifying vehicle loads on bridge decks based on high-altitude aerial images

Also Published As

Publication number Publication date
CN114187219B (en) 2024-06-25

Similar Documents

Publication Publication Date Title
KR101518722B1 (en) Image enhancement processing method and apparatus for distortion correction by air particle like fog
CN108805023B (en) Image detection method, device, computer equipment and storage medium
US8861851B2 (en) Color highlight reconstruction
CN105956539B (en) A kind of Human Height measurement method of application background modeling and Binocular Vision Principle
CN108830800B (en) A method for enhancing the brightness of images in dark scenes
CN102385753B (en) An Adaptive Image Segmentation Method Based on Illumination Classification
CN102768760B (en) Quick image dehazing method on basis of image textures
JP2006350557A5 (en)
WO2017215527A1 (en) Hdr scenario detection method, device, and computer storage medium
CN102867295B (en) A kind of color correction method for color image
CN105701785B (en) The image haze minimizing technology of Weighted T V transmissivities optimization is divided based on sky areas
CN106600569B (en) Signal lamp color effect enhancement processing method and device
CN109544583A (en) A kind of method, device and equipment for extracting Leather Image area-of-interest
CN107481214A (en) A kind of twilight image and infrared image fusion method
CN114187219A (en) Moving target shadow real-time elimination method based on red, green and blue double difference
CN110765887A (en) Automatic identification technology and detection method for tunnel lining cracks
CN104751487B (en) A kind of moving target detecting method based on the plane discoloration frames of colored RGB tri- difference
CN111080722B (en) Color migration method and system based on significance detection
CN112668426A (en) Fire disaster image color cast quantization method based on three color modes
CN103985089B (en) With reference to weight edge analysis and the image streak correction method of frame inner iteration
CN108010050A (en) A kind of foreground detection method based on adaptive RTS threshold adjustment and selective context update
Li et al. A perceptual color edge detection algorithm
CN109191481B (en) A fast threshold segmentation method for shoe print images under bad exposure conditions
CN104992418B (en) A kind of abnormal color bearing calibration suitable for thermal imagery video color
CN108389219B (en) Weak and small target tracking loss re-detection method based on multi-peak judgment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant