CN112906733B - Two-stage double-check bounded-bias correlation real-time template matching method - Google Patents

Two-stage double-check bounded-bias correlation real-time template matching method Download PDF

Info

Publication number
CN112906733B
CN112906733B CN202110025056.2A CN202110025056A CN112906733B CN 112906733 B CN112906733 B CN 112906733B CN 202110025056 A CN202110025056 A CN 202110025056A CN 112906733 B CN112906733 B CN 112906733B
Authority
CN
China
Prior art keywords
matching
template
image
point
matched
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110025056.2A
Other languages
Chinese (zh)
Other versions
CN112906733A (en
Inventor
陈逢军
廖金麒
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hunan University
Original Assignee
Hunan University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hunan University filed Critical Hunan University
Priority to CN202110025056.2A priority Critical patent/CN112906733B/en
Publication of CN112906733A publication Critical patent/CN112906733A/en
Application granted granted Critical
Publication of CN112906733B publication Critical patent/CN112906733B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/22—Matching criteria, e.g. proximity measures

Abstract

The invention discloses a real-time template matching method of two-stage double-check bounded-bias correlation, belonging to the digital image processing technology. The method solves the problems of high calculation complexity and long matching time of the traditional normalized product correlation template matching algorithm, thereby realizing the template matching with real-time property. Firstly, calculating an optimal sampling factor according to the sizes of a scene image to be matched and a template image; secondly, simultaneously downsampling the scene image and the template image according to the optimal sampling factor; then carrying out coarse matching on the sampled scene image and template image by using a double-check bounded-bias correlation template matching algorithm to obtain an optimal coarse matching point; and finally, mapping the rough matching point to a scene image to be matched before sampling, and performing fine matching by using a double-check bounded partial correlation template matching algorithm again to obtain the optimal matching point.

Description

Two-stage double-check bounded partial correlation real-time template matching method
Technical Field
The invention relates to a real-time template matching method of two-stage double-check bounded-bias correlation, belonging to the field of industrial real-time image matching.
Background
Template matching refers to a technical process of finding a given reference template position in a target scene or target image. The core technique of template matching, which is a fundamental task in countless image analysis applications, is the algorithm used for template matching. Generally, the method can be divided into gray-level-based template matching and feature-based template matching, the feature-based template matching algorithm is complex in operation and long in time consumption, and the algorithm hardly meets the real-time requirement, so that more gray-level-based template matching algorithms are adopted in practical application.
The gray-scale-based template matching is to slide a template to a search area, calculate a distortion degree or a correlation degree at each position, measure the similarity degree or the difference degree between the template and a subgraph, then represent the position of an object in a matched image by adopting the position with the minimum distortion degree or the maximum similarity degree, and simultaneously perform threshold setting on the similarity degree or the distortion degree according to the application requirements to allow the rejection of matching which does not meet the threshold requirement. Typical algorithms include an absolute error sum algorithm and an error square sum algorithm, and a normalized product correlation algorithm is the most widely applied correlation measurement algorithm at present due to good anti-interference performance of the normalized product correlation algorithm. However, the normalized product algorithm has too large calculation amount to meet the real-time positioning application in industry.
In order to solve the problems, the invention provides a two-stage double-check bounded-offset correlated real-time template matching method, which accelerates a normalized product algorithm and simultaneously realizes coarse matching and fine matching by using a sampling technology of variable sampling factors so as to achieve industrial real-time template matching positioning.
Disclosure of Invention
The method aims at the characteristics of high calculation complexity and large calculation amount of the traditional template matching algorithm related to normalized products. The invention provides a real-time template matching method of two-stage double-check bounded partial correlation on the basis of a template matching algorithm of normalized product correlation, solves the problem of compatibility between the theory and practical application of the template matching algorithm, and makes industrial real-time positioning possible. The invention discloses a real-time template matching method of two-stage double-check bounded partial correlation, which comprises the following steps:
the method comprises the following steps: calculating an optimal sampling factor according to the sizes of the scene image to be matched and the template image:
step two: carrying out down-sampling of an optimal sampling factor on a scene image to be matched and a template image to obtain a sampled scene image and template image;
step three: performing coarse matching on the sampled image by using a double check bounded partial correlation (DBPC) template matching algorithm to obtain an optimal coarse matching point;
step four: and mapping the optimal matching point obtained by coarse matching into the scene image to be matched before sampling, and performing fine matching in the neighborhood of 4factor multiplied by 4factor around the mapping point by using a DBPC template matching algorithm to obtain the optimal matching point, namely the position of the template in the scene image to be matched.
The invention discloses a real-time template matching method of two-stage double-check bounded partial correlation, which is characterized in that the optimal sampling factor in the first step is as follows:
Figure BDA0002889953010000021
wherein, W and H are the width and height of the scene image to be matched respectively, and M and N are the width and height of the template image respectively; the first term of equation (1) represents the total number of operations required to perform downsampling of the template and the image to be matched, while the second term represents the total number of operations required to perform matching in a 4k × 4k neighborhood of the image to be matched.
The invention discloses a real-time template matching method of two-stage double-check bounded-bias correlation, which is characterized in that a DBPC algorithm in the third step and the fourth step specifically comprises the following steps:
step 1: selecting initialization threshold eta and correlation ratio parameter pair (C)r1,Cr2) And calculating | T |)2And the following formula:
Figure BDA0002889953010000022
Figure BDA0002889953010000023
Figure BDA0002889953010000024
Figure BDA0002889953010000025
wherein n is1Is the number of rows where the first partition boundary is located, n2Is the number of rows (n) at which the boundary of the second partition is located2>n1)，|T|2L representing a template image T2Norm, beta1-1，β1-2，β1-3Respectively representing L that are to be matching subgraphs2Norm | I (x, y) & gtluminance2According to n1And n2The calculation was performed in three regions as shown in fig. 1. And (| I (x, y) & gtnon |)2)2＝β1-1+β1-2+β1-3。
Step 2: calculating the first upper boundary term alpha1(x,y)：
Figure BDA0002889953010000026
Step 3: judging the first checking condition of the formula (7), if yes, skipping the current point, and switching to the next point judgment; if not, calculating a second upper boundary term alpha by formula (8)2(x, y); then, judging a second checking condition of the formula (9), if yes, skipping the current point, and switching to the judgment of the next point; if not, calculating the actual similarity NCC (x, y), namely the formula (10);
Figure BDA0002889953010000031
Figure BDA0002889953010000032
Figure BDA0002889953010000033
Figure BDA0002889953010000034
step 4: judging the formula (11), if so, judging that the current point is not the optimal matching point; if not, the similarity of the current point is greater than eta, the current similarity value is assigned to eta, and the position of the current matching point is recorded;
NCC(x,y)＜η (11)
step 5: and traversing the whole image to be matched, and repeating the processes from Step2 to Step4 until the most similar matching point is found.
The core DBPC algorithm in the two-stage double-check bounded partial correlation real-time template matching method is characterized in that the initialization performance parameters in Step1 can be obtained by utilizing experimental tests; the method can also be automatically obtained by the following method:
I. the method for automatically determining the initialization threshold η is as follows: aiming at different conditions, uniformly distributing or Gaussian distributing in a search space, and generating random points with fixed quantity and size by using an MWC (multi-way-carry multiplication) algorithm, wherein when the search space is more than million pixels, the number of the generated random points can be determined according to 0.001-0.005 of the search space; if the search space is below megapixels, the number of generated random points can be determined according to 0.01-0.05 of the search space; as shown in FIG. 2, let each random point location be (x)i,yi) Then the initialization threshold is determined as follows:
Figure BDA0002889953010000035
wherein n represents the total number of random points;
II. Correlation ratio parameter pair (C)r1,Cr2) Is chosen to be (0.2, 0.4).
The real-time template matching method of the two-stage double-check bounded-bias correlation is characterized in that an efficiency measurement standard irrelevant to hardware is provided and used for measuring the efficiency measurement between a DBPC template matching algorithm and an original normalized product correlation template matching algorithm; the ratio of the total number of the points to be eliminated to the total number of the points to be matched by the check condition is defined as an elimination ratio, and the two elimination ratios are expressed as follows:
Er1＝S1/P，Er2＝S2/P (13)
wherein S is1The formula (7) is independently used as a check condition, and the total number of the skipping points is calculated when the condition is met; s2The formula (9) is independently used as a check condition, the total number of the skipping points meeting the condition is shown, and P represents the total number of the matching points during actual searching;
then, the efficiency metric R2Is defined as follows：
Figure BDA0002889953010000041
Has the advantages that:
the invention provides a bounded partial correlation real-time template matching method with two-stage double-check, which is mainly characterized by a variable sampling factor strategy, an automatic determination initialization performance parameter strategy and a rough matching and fine matching search strategy, solves the non-real-time matching problem based on an NCC template matching algorithm, and realizes industrial vision real-time positioning.
Drawings
FIG. 1: partitioning rules based on consecutive rows.
FIG. 2: two methods for automatically determining the initialization threshold eta; (a) a schematic diagram of uniform distribution; (b) schematic representation of gaussian distribution.
FIG. 3: the whole two-stage double-check bounded partial correlation (TDBPC)) template matching method is a flow chart.
FIG. 4: a flow chart of a double check bounded partial correlation (DBPC) template matching method.
FIG. 5: 4 PCB circuit board images with different resolution ratios and corresponding template images.
FIG. 6: and 6 groups of PCB circuit board images with different resolution ratios and corresponding template images.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more clear, embodiments of the present invention are further described in detail below with reference to the accompanying drawings:
as shown in fig. 3, this embodiment provides a real-time template matching method for two-stage double-check bounded partial correlation, where the method includes:
the method comprises the following steps: calculating an optimal sampling factor according to the sizes of the scene image to be matched and the template image;
step two: carrying out down-sampling of an optimal sampling factor on a scene image and a template image to be matched to obtain a sampled scene image and template image;
step three: carrying out coarse matching on the sampled images by using a double check bounded bias correlation (DBPC) matching algorithm to obtain an optimal coarse matching point;
step four: and mapping the optimal matching point obtained by coarse matching to a scene image to be matched before sampling, and performing fine matching in a 4factor multiplied by 4factor neighborhood around the mapping point by using a DBPC template matching algorithm again to obtain the optimal matching point, namely the position of the template in the scene image to be matched.
As shown in fig. 4, the DBPC algorithm mentioned in step three and step four includes:
step 1: selecting initialization threshold eta and correlation ratio parameter pair (C)r1,Cr2) And calculating | T |)2And the values of equations (3), (4), (5);
step 2: calculating a first upper bound term α by equation (6)1(x,y)；
Step 3: judging the first check condition of the formula (7), if yes, skipping the current point and judging the next point; if not, calculating a second upper boundary term alpha by formula (8)2(x, y); then, judging a second checking condition of the formula (9), if so, skipping the current point and judging the next point; if not, calculating the actual similarity NCC (x, y);
step 4: judging the formula (11), if so, judging that the current point is not the optimal matching point; if not, the similarity of the current point is greater than eta, the current similarity value is assigned to eta, and the position of the current matching point is recorded;
step 5: and traversing the whole image to be matched, and repeating the processes from Step2 to Step4 until the most similar matching point is found.
Example 1
The four images with different resolutions and different matching targets shown in fig. 5 are used for influencing the algorithm matching efficiency by the algorithm performance parameters; as can be seen from equations (7) and (9), the initialization threshold η is the first performance parameter of the double-check bounded partial correlation algorithm. Retention of Cr1＝0.4，Cr2When the value is 0.2 unchanged, eta is 0.94, 0.95, 0.96 and 0.97 respectively, and the matching effect is calculatedRate metric index R2。
TABLE 1 η Pair Algorithm matching efficiency metric R2Influence of (2)
Figure BDA0002889953010000051
In Table 1, all images R2<And 50%, which shows that the calculation amount of the double-check bounded partial correlation method provided by the invention can be reduced by more than half compared with the standard NCC (x, y) function. As η increases, R2The matching efficiency is increased as the result of gradual reduction; r of Image22<22%, indicating that the calculated amount is reduced by more than 78%; r of Image42<31%, indicating a reduction in the calculated amount of more than 69%. In maintenance of Cr1And Cr2Under the precondition of unchanging, R2The smaller the matching efficiency.
Example 2
The algorithm matches the efficiency metric R2Parameter pair with correlation ratio (C)r1,Cr2) Are related to discrete variables due to 0<Cr1<Cr2<1, when C isr2When the average value is 0.2 to 0.9, Cr1From 0.1 to Cr2The variation is carried out with a step size of 0.1, and 36 sets of parameter pairs are obtained. And obtaining a reference parameter pair after experimental tests are carried out, and then selecting one parameter pair for actual template matching. The specific implementation method comprises the following steps:
using the above 36 sets of parameters, 60 images of different sizes were tested, each image yielding 36R2Sorting the data in a descending order, selecting the parameter pairs corresponding to the first 7 data from each picture, and selecting 7 parameter pairs to be the parameter pairs to be referred to according to the frequency; table 2 shows that the sum of the number of occurrences of the first 7 parameter pairs in 420 data exceeds 60% of the total number of data, and these 7 parameter pairs can be used as the parameter pairs to be referred to.
TABLE 2.60 test results for images
Figure BDA0002889953010000061
Example 3
Simulating an actual stable scene under a natural light scene; the 66 images were collected and divided into 6 sub-groups of different resolution sizes, where one image in each sub-group was used for the source of the template and 10 additional images were used for testing. The information for each set of images is shown in table 4 below.
Initialization parameters are first determined and the first image of each group is selected, the image and template being as shown in fig. 6. A suitable initialization threshold η is chosen. Using a two-stage template matching algorithm to test 7 parameter pairs to be selected so as to measure a matching efficiency metric R in a coarse matching process2Selecting an optimal parameter pair for the metric; the matching results for the 6 sets of images are shown in table 3 below:
TABLE 3 seven parameters vs R tested2Data and parameter selection
Figure BDA0002889953010000071
Table 3 shows that the initialization parameters for 6 sets of test images can be selected.
The test platform is a Window platform with a CORE i5 processor of Intel with a main frequency of 2.3GHz, and the memory is 4G. And c + + is used for running the algorithm program in the compiling environment of VS 2017. The two-stage double-check bounded partial correlation method provided by the invention is used for testing, meanwhile, the traditional NCC template matching algorithm is used for comparison, 6 groups of images are respectively tested, the running time of the algorithm in the testing process is taken as the measurement standard of the matching efficiency, and the testing results in a table 4 show that: the average time consumption of the method is lower than that of the traditional algorithm.
TABLE 4 six test data the average elapsed time (in ms) for the present invention and the conventional algorithm
Figure BDA0002889953010000072

Claims (3)

1. A real-time template matching method for two-stage double-check bounded partial correlation is characterized by comprising the following steps:
the method comprises the following steps: calculating an optimal sampling factor according to the sizes of the scene image to be matched and the template image, specifically:
Figure FDA0003567711530000011
w and H are the width and the height of a scene image to be matched respectively, M and N are the width and the height of a template image respectively, a first item of the formula (1) represents the total number of operations required for performing downsampling on the template image and the image to be matched, and a second item represents the total number of operations required for performing matching in a 4k multiplied by 4k neighborhood of the image to be matched;
step two: carrying out down-sampling of an optimal sampling factor on a scene image to be matched and a template image to obtain a sampled scene image and template image;
step three: performing coarse matching on the sampled image by using a double check bounded partial correlation template matching algorithm, namely a DBPC template matching algorithm to obtain an optimal coarse matching point;
step four: mapping the optimal matching point obtained by coarse matching to a scene image to be matched before sampling, and performing fine matching in a 4factor multiplied by 4factor neighborhood around the mapping point by using a DBPC template matching algorithm again to obtain the optimal matching point, namely the position of the template in the scene image to be matched;
the DBPC template matching algorithm in the third step and the fourth step specifically comprises the following steps:
step 1: selecting initialization threshold eta and correlation ratio parameter pair (C)r1,Cr2) And calculating | T |)2And the following formula:
Figure FDA0003567711530000012
Figure FDA0003567711530000013
Figure FDA0003567711530000014
Figure FDA0003567711530000015
wherein n is1Is the number of rows where the first partition boundary is located, n2Is the number of rows (n) where the second partition boundary is located2>n1)，|T|2Is L of the template image T2Norm, beta1-1，β1-2，β1-3Respectively representing L that are to be matching subgraphs2Norm | I (x, y) & gtluminance2According to n1And n2Divided into three regions for calculation, and (| I (x, y) & gtdoes not count2)2＝β1-1+β1-2+β1-3；
Step 2: calculating a first upper bound term alpha1(x,y)：
Figure FDA0003567711530000016
Step 3: judging the first check condition of the formula (7), if yes, skipping the current point and judging the next point; if not, calculating a second upper boundary term alpha by formula (8)2(x, y), then judging a second checking condition of the formula (9), if yes, skipping the current point, and judging the next point; if not, calculating the actual similarity NCC (x, y), namely the formula (10);
Figure FDA0003567711530000021
Figure FDA0003567711530000022
Figure FDA0003567711530000023
Figure FDA0003567711530000024
step 4: judging the formula (11), if so, judging that the current point is not the optimal matching point; if not, the similarity of the current point is greater than eta, the current similarity value is assigned to eta, and the position of the current matching point is recorded;
NCC(x,y)＜η (11)
step 5: and traversing the whole image to be matched, and repeating the processes from Step2 to Step4 until the most similar matching point is found.
2. The method of claim 1, wherein the DBPC template matching algorithm is characterized by the fact that the initialization performance parameters in Step1 can be obtained by experimental tests; the method can also be automatically obtained by the following method:
I. the method for automatically determining the initialization threshold η is as follows: aiming at different conditions, random points with fixed quantity and size are generated in a search space by adopting uniform distribution or Gaussian distribution and utilizing a carry multiplication algorithm, and when the search space is more than million pixels, the number of the generated random points is determined according to 0.001-0.005 of the search space; if the search space is below megapixels, the number of generated random points is determined according to 0.01-0.05 of the search space; let each random point location be (x)i,yi) Then the initialization threshold is determined as follows:
Figure FDA0003567711530000025
wherein n represents the total number of random points;
II. Correlation ratio parameter pair (C)r1,Cr2) Is chosen to be (0.2, 0.4).
3. The method of claim 1, wherein the DBPC template matching algorithm provides a hardware-independent efficiency metric for measuring an efficiency metric between the DBPC template matching algorithm and an original normalized product-dependent template matching algorithm; the ratio of the total number of the points to be eliminated to the total number of the points to be matched by the check condition is defined as an elimination ratio, and the two elimination ratios are expressed as follows:
Er1＝S1/P，Er2＝S2/P (13)
wherein S is1The formula (7) is independently used as a check condition, and the total number of the skipping points is calculated when the condition is met; s2The formula (9) is independently used as a check condition, the total number of the skipping points meeting the condition is shown, and P represents the total number of the matching points during actual searching;
then, the efficiency metric R2The definition is as follows:
Figure FDA0003567711530000031
CN202110025056.2A 2021-01-08 2021-01-08 Two-stage double-check bounded-bias correlation real-time template matching method Active CN112906733B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110025056.2A CN112906733B (en) 2021-01-08 2021-01-08 Two-stage double-check bounded-bias correlation real-time template matching method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110025056.2A CN112906733B (en) 2021-01-08 2021-01-08 Two-stage double-check bounded-bias correlation real-time template matching method

Publications (2)

Publication Number Publication Date
CN112906733A CN112906733A (en) 2021-06-04
CN112906733B true CN112906733B (en) 2022-06-03

Family

ID=76112294

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110025056.2A Active CN112906733B (en) 2021-01-08 2021-01-08 Two-stage double-check bounded-bias correlation real-time template matching method

Country Status (1)

Country Link
CN (1) CN112906733B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113256723B (en) * 2021-06-29 2023-03-21 西南交通大学 Automatic detection method for pantograph lifting time and pantograph head displacement curve

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102194115A (en) * 2010-03-18 2011-09-21 富士通株式会社 Image processing apparatus and image processing method
CN103617625A (en) * 2013-12-13 2014-03-05 中国气象局北京城市气象研究所 Image matching method and image matching device
CN103913464A (en) * 2014-03-20 2014-07-09 湖南大学 High speed railway track surface defect coupling method based on machine visual inspection
CN112149671A (en) * 2020-09-28 2020-12-29 天津大学 Target area accurate positioning method based on normalized cross-correlation template matching

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102194115A (en) * 2010-03-18 2011-09-21 富士通株式会社 Image processing apparatus and image processing method
CN103617625A (en) * 2013-12-13 2014-03-05 中国气象局北京城市气象研究所 Image matching method and image matching device
CN103913464A (en) * 2014-03-20 2014-07-09 湖南大学 High speed railway track surface defect coupling method based on machine visual inspection
CN112149671A (en) * 2020-09-28 2020-12-29 天津大学 Target area accurate positioning method based on normalized cross-correlation template matching

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
A_Facial_Expression_Recognition_Algorithm_Based_on_Feature_Fusion;chen fengjun;《IEEE：A_Facial_Expression_Recognition_Algorithm_Based_on_Feature_Fusion》;20081231;全文 *
基于机器视觉的LED划片机定位技术研究;叶小奇;《中国知网》;中国知网;20190131;全文 *
模板匹配技术在玫瑰扫描亚图像目标识别中的应用;邹立建等;《光学技术》;20010320(第02期);全文 *

Also Published As

Publication number Publication date
CN112906733A (en) 2021-06-04

Similar Documents

Publication Publication Date Title
CN110349126B (en) Convolutional neural network-based marked steel plate surface defect detection method
CN111145175B (en) SMT welding spot defect detection method based on iForest model verification
CN109509187B (en) Efficient inspection algorithm for small defects in large-resolution cloth images
CN105786860B (en) Data processing method and device in data modeling
CN109934810B (en) Defect classification method based on improved particle swarm wavelet neural network
CN106778734A (en) A kind of insulator based on rarefaction representation falls to go here and there defect inspection method
CN110689524B (en) No-reference online image definition evaluation method and system
CN108664839B (en) Image processing method and device
CN112258470B (en) Intelligent industrial image critical compression rate analysis system and method based on defect detection
CN109685097A (en) A kind of image detecting method and device based on GAN
CN112906733B (en) Two-stage double-check bounded-bias correlation real-time template matching method
CN114998314A (en) PCB (printed Circuit Board) defect detection method based on computer vision
CN112907561A (en) Notebook appearance flaw detection method based on deep learning
CN115082477B (en) Semiconductor wafer processing quality detection method based on light reflection removing effect
CN111415353A (en) Detection structure and detection method for fastener burr defects based on ResNet58 network
CN112819748A (en) Training method and device for strip steel surface defect recognition model
CN114612345A (en) Light source detection method based on image processing
CN112381751A (en) Online intelligent detection system and method based on image processing algorithm
CN116188419A (en) Lightweight cloth flaw detection method capable of being deployed in embedded equipment
CN110544249A (en) Convolutional neural network quality identification method for arbitrary-angle case assembly visual inspection
CN113176270B (en) Dimming method, device and equipment
CN111476750A (en) Method, device and system for carrying out stain detection on imaging module and storage medium
RU2458397C1 (en) Method of searching for and recognising objects on digital images
CN111667509A (en) Method and system for automatically tracking moving target under condition that target is similar to background color
CN115861305A (en) Flexible circuit board detection method and device, computer equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant