CN110245671A - A kind of endoscopic images characteristic point matching method and system - Google Patents
A kind of endoscopic images characteristic point matching method and system Download PDFInfo
- Publication number
- CN110245671A CN110245671A CN201910521138.9A CN201910521138A CN110245671A CN 110245671 A CN110245671 A CN 110245671A CN 201910521138 A CN201910521138 A CN 201910521138A CN 110245671 A CN110245671 A CN 110245671A
- Authority
- CN
- China
- Prior art keywords
- characteristic point
- point
- matching
- endoscopic images
- matched
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 54
- 230000002146 bilateral effect Effects 0.000 claims abstract description 42
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims abstract description 29
- 230000003044 adaptive effect Effects 0.000 claims abstract description 25
- 230000008447 perception Effects 0.000 claims abstract description 11
- 238000012545 processing Methods 0.000 claims description 22
- 230000006870 function Effects 0.000 claims description 13
- 238000013459 approach Methods 0.000 claims description 8
- 239000000284 extract Substances 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 3
- 230000001052 transient effect Effects 0.000 claims description 3
- 239000011521 glass Substances 0.000 claims description 2
- 238000004422 calculation algorithm Methods 0.000 description 19
- 238000010586 diagram Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 7
- 238000004891 communication Methods 0.000 description 6
- 238000007634 remodeling Methods 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000000717 retained effect Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 230000007423 decrease Effects 0.000 description 2
- 230000007812 deficiency Effects 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 238000001356 surgical procedure Methods 0.000 description 2
- 230000000740 bleeding effect Effects 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000002324 minimally invasive surgery Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 231100000241 scar Toxicity 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 230000008961 swelling Effects 0.000 description 1
- 239000013598 vector Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/136—Segmentation; Edge detection involving thresholding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V10/757—Matching configurations of points or features
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
- Endoscopes (AREA)
Abstract
The embodiment of the present invention provides a kind of endoscopic images characteristic point matching method and system.This method includes extracting image characteristic point based on two endoscopic images to be matched, and complete Image Feature Point Matching based on Feature Descriptor similarity, obtains initial matching point to set；Local distance constraint is carried out to set to initial matching point, characteristic point information is extended in conjunction with affine parameter and character pair point motion information, similarity boundary is estimated, obtains the characteristic point correspondence set with Movement consistency；Space length perception is optimized based on characteristic point correspondence set, bilateral affine motion consistency model is generated, the adaptive distance threshold parameters of bilateral moving boundaries is set, the corresponding interior point matching set of global image is obtained, realizes Feature Points Matching.The embodiment of the present invention finds reliable corresponding relationship, and guaranteeing high-precision while retaining enough Feature Points Matchings pair by the Movement consistency method kept based on locality from given initial matching pair.
Description
Technical field
The present invention relates to field of image processing more particularly to a kind of endoscopic images characteristic point matching method and systems.
Background technique
Endoscope is a kind of optical instrument, is shown by cold light source camera lens, fiber optic conducting wire, image delivering system, screen
The composition such as system, it can expand surgical field of view, and the outstanding feature using endoscope is that operative incision is small, and incisional scar is unobvious,
After-operation response is light, and bleeding, livid purple and swelling time can greatly reduce, and restores also fast compared with traditional operation.Passing through area of computer aided
Medical endoscopes field of surgery, it can be common that carry out associated picture processing to guide Minimally Invasive Surgery, build between images
Founding reliable corresponding relationship is the critical issue in numerous clinical applications, for example, tissue surface rebuild, camera motion estimation with
And in the application such as surgical navigational.
Existing endoscopic images characteristic point matching method is generally special using common detection and description image local first
Algorithm is levied, such as utilizes affine scale invariant feature conversion method (Affine Scale Invariant Feature
Transform, hereinafter referred to as ASIFT) etc. extract the characteristic point of image, and using corresponding Feature Descriptor come to characteristic point
Match the feature point correspondence between available image, but due to the general texture of institutional framework in endoscopic images
Information is weak, exists and such as blocks and the problems such as deformation, simple dependence characteristics describe sub- similarity be difficult to obtain it is satisfactory
Matching result.
Existing endoscopic images characteristic point matching method is primarily present following problem: extracting characteristic point in the case of weak texture
Number deficiency, the matching result lazy weight finally retained are difficult to realize careful surface or structural remodeling；Change threshold value to mention
When taking fully measure feature point, since texture information is weak, there are factors such as deformation, the matching number of mistake is more, a large amount of outlier without
Method is effectively rejected, and matching precision is caused to decline.
For clinical medicine image procossing, obtains sufficient amount and high-precision matching result is that image guidance is minimally invasive
The key of successful surgery, it is therefore desirable to a kind of method for proposing high-precision endoscopic images Feature Points Matching.
Summary of the invention
The embodiment of the present invention provides a kind of endoscopic images characteristic point matching method and system, to solve in the prior art
Image characteristic point extracts lazy weight, and matching precision is low, the problems such as influence by a large amount of interference noise points.
In a first aspect, the embodiment of the present invention provides a kind of endoscopic images characteristic point matching method, comprising:
S1 extracts image characteristic point based on two endoscopic images to be matched, and complete based on Feature Descriptor similarity
At described image Feature Points Matching, the initial matching point of described image characteristic point is obtained to set；
S2, based on the unknown deformation and described image characteristic point local neighborhood between two endoscopic images to be matched
Corresponding relationship and partial structurtes between structure carry out local distance constraint to set to the initial matching point, in conjunction with affine
Parameter and character pair point motion information are extended characteristic point information, estimate similarity boundary, obtain having movement consistent
The characteristic point correspondence set of property；
S3 optimizes space length perception based on the characteristic point correspondence set, generates bilateral affine motion
The adaptive distance threshold parameters of bilateral moving boundaries are arranged in consistency model, obtain the corresponding interior set of matches of global image
It closes, realizes the Feature Points Matching between two endoscopic images to be matched.
Wherein, the step of S2 is specifically included:
S21 establishes the unknown deformation between two endoscopic images to be matched and described image characteristic point part neighbour
Corresponding relationship between domain structure；
S22 is based on each described image characteristic point, and corresponding 6 closest approaches construct the office in corresponding point set
Portion's structure；
S23, based between two endoscopic images to be matched unknown deformation and described image characteristic point part it is adjacent
Corresponding relationship between domain structure, and any the distance between the initial matching point pair and its neighborhood point are fixed, and threshold value is arranged
Parameter calculates and obtains the interior point set kept with locality, realizes local distance constraint；
S24, the characteristic point information extracted to the two images to be matched are extended, and addition characteristic point is corresponding
Motion information obtain the characteristic point correspondence set with the Movement consistency using similarity boundary function.
Wherein, the step S22 includes being calculated based on Euclidean distance formula, is obtained based on each described image
Characteristic point and its in corresponding point set corresponding 6 closest approaches come the partial structurtes that construct.
Wherein, similarity boundary function is applied in the step S24, obtains the characteristic point pair with Movement consistency
Set of relationship is answered, is based on obtained by the interior point set kept with locality.
Wherein, the step of S3 is specifically included:
S31 is based on the characteristic point correspondence set, using the affine motion boundary of bilateral variation, obtains described double
Side affine motion consistency model；
The adaptive space threshold parameter of the bilateral moving boundaries is arranged in S32, and it is corresponding interior to obtain the global image
Point matching set, realizes the Feature Points Matching between two endoscopic images to be matched.
Wherein, the adaptive space threshold parameter of the bilateral moving boundaries is set in the step S32, is specifically included:
Model is kept in conjunction with partial structurtes, sets the adaptive space threshold parameter of the bilateral moving boundaries to and institute
The distance restraint threshold parameter for stating partial structurtes is consistent.
Wherein, the corresponding interior point matching set of the global image is calculated and obtained in the step S32, is specifically included:
It is special by the way that estimation result and described image of the described image characteristic point in bilateral moving boundaries both direction is arranged
The distance between sign spot noise observation data threshold value is gathered to obtain the corresponding interior point matching of the global image.
Second aspect, the embodiment of the present invention provide a kind of endoscopic images Feature Points Matching system, comprising:
First processing module for extracting image characteristic point based on two endoscopic images to be matched, and is based on feature
It describes sub- similarity and completes described image Feature Points Matching, obtain the initial matching point of described image characteristic point to set；
Second processing module, for based on the unknown deformation and described image between two endoscopic images to be matched
Corresponding relationship and partial structurtes between characteristic point local neighborhood structure to the initial matching point to set carry out part away from
From constraint, characteristic point information is extended in conjunction with affine parameter and character pair point motion information, similarity boundary is estimated, obtains
To the characteristic point correspondence set with Movement consistency；
Third processing module, it is raw for being optimized based on the characteristic point correspondence set to space length perception
At bilateral affine motion consistency model, the adaptive distance threshold parameters of bilateral moving boundaries are set, global image pair is obtained
The interior point matching set answered, realizes the Feature Points Matching between two endoscopic images to be matched.
The third aspect, the embodiment of the present invention provide a kind of electronic equipment, comprising:
Memory, processor and storage on a memory and the computer program that can run on a processor, the processing
Device realizes a kind of the step of any one endoscopic images characteristic point matching method when executing described program.
Fourth aspect, the embodiment of the present invention provide a kind of non-transient computer readable storage medium, are stored thereon with calculating
Machine program any one of realizes a kind of endoscopic images characteristic point matching method when computer program is executed by processor
Step.
A kind of endoscopic images characteristic point matching method provided in an embodiment of the present invention and system, by being protected based on locality
The Movement consistency method held, finds reliable corresponding relationship from the given initial matching pair comprising a large amount of outliers, and
Guarantee high-precision while retaining enough Feature Points Matchings pair, helps to realize finer tissue surface and rebuild, is more quasi-
The application such as true camera motion estimation and surgical navigational.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is the present invention
Some embodiments for those of ordinary skill in the art without creative efforts, can also basis
These attached drawings obtain other attached drawings.
Fig. 1 is a kind of endoscopic images characteristic point matching method flow chart provided in an embodiment of the present invention；
Fig. 2 is step S2 detailed process in a kind of endoscopic images characteristic point matching method provided in an embodiment of the present invention
Figure；
Fig. 3 is step S3 detailed process in a kind of endoscopic images characteristic point matching method provided in an embodiment of the present invention
Figure；
Fig. 4 is a kind of endoscopic images Feature Points Matching system structure diagram provided in an embodiment of the present invention；
Fig. 5 is a kind of Second processing module sub- knot of endoscopic images Feature Points Matching system provided in an embodiment of the present invention
Structure schematic diagram；
Fig. 6 is a kind of third processing module sub- knot of endoscopic images Feature Points Matching system provided in an embodiment of the present invention
Structure schematic diagram；
Fig. 7 is matching algorithm flow chart provided in an embodiment of the present invention；
Fig. 8 is the Movement consistency bounding algorithm flow chart provided in an embodiment of the present invention kept based on partial structurtes；
Fig. 9 is provided in an embodiment of the present invention based on the bilateral moving boundaries bounding algorithm flow chart of adaptive threshold；
Figure 10 is the structural block diagram of electronic equipment provided in an embodiment of the present invention.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with the embodiment of the present invention
In attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is
A part of the embodiment of the present invention, instead of all the embodiments.Based on the embodiments of the present invention, those of ordinary skill in the art
Every other embodiment obtained without making creative work, shall fall within the protection scope of the present invention.
There are the following problems for existing endoscopic images characteristic point matching method: characteristic point number is extracted in the case of weak texture
Deficiency, the matching result lazy weight finally retained are difficult to realize careful surface or structural remodeling；Change threshold value and extracts foot
When measure feature point, since texture information is weak, there are factors such as deformation, the matching number of mistake is more, and a large amount of outlier can not be by
It effectively rejects, matching precision is caused to decline.
Aiming at the problems existing in the prior art, the embodiment of the invention provides a kind of endoscopic images Feature Points Matching sides
Method, Fig. 1 are a kind of endoscopic images characteristic point matching method flow chart provided in an embodiment of the present invention, as shown in Figure 1, comprising:
S1 extracts image characteristic point based on two endoscopic images to be matched, and complete based on Feature Descriptor similarity
At described image Feature Points Matching, the initial matching point of described image characteristic point is obtained to set；
S2, based on the unknown deformation and described image characteristic point local neighborhood between two endoscopic images to be matched
Corresponding relationship and partial structurtes between structure carry out local distance constraint to set to the initial matching point, in conjunction with affine
Parameter and character pair point motion information are extended characteristic point information, estimate similarity boundary, obtain having movement consistent
The characteristic point correspondence set of property；
S3 optimizes space length perception based on the characteristic point correspondence set, generates bilateral affine motion
The adaptive distance threshold parameters of bilateral moving boundaries are arranged in consistency model, obtain the corresponding interior set of matches of global image
It closes, realizes the Feature Points Matching between two endoscopic images to be matched.
Specifically, two endoscopic images characteristic points to be matched are extracted first in step sl, it is assumed that in one pair given
Sight glass image is respectively Ir(reference picture) and It(template image), using image characteristics extraction algorithm, the present embodiment is used
ASIFT algorithm, the feature extracted in two images respectively obtain character pair point set FrAnd Ft.Due to group in endoscopic images
It knits that smooth, texture information is weak, therefore to consider all possible matching result, obtain one using the similarity of Feature Descriptor
Initial feature point correspondence, is defined asPurpose is in order to by rejecting SoIn outlier
To obtain reliable corresponding relationship.Unknown deformation between two endoscopic images to be matched and described is initially set up in step S2
Corresponding relationship between image characteristic point local neighborhood structure, and based on above-mentioned corresponding relationship and partial structurtes to being obtained in step S1
The initial matching point taken carries out local distance constraint to set, then introduces affine parameter and the movement of character pair point to characteristic point
Information is extended, then estimates similarity boundary to realize the consistency constraint to characteristic point movement.Step S3 is in step s 2
Space length perception is optimized on the basis of all set of characteristic points for meeting similarity boundary generated, is obtained bilateral imitative
Movement consistency model is penetrated, while using the average distance of partial structurtes as the adaptive apart from threshold of bilateral moving boundaries model
Value, carries out judgement calculating, it is therefore an objective to reject the noise outlier in image, and obtain final Feature Points Matching result.
Whole matching algorithm process is as shown in fig. 7, Fig. 7 is matching algorithm flow chart provided in an embodiment of the present invention.
The embodiment of the present invention is by the Movement consistency method that is kept based on locality, from given comprising a large amount of outliers
Reliable corresponding relationship is found in initial matching pair, and is being guaranteed high-precision while being retained enough Feature Points Matchings pair,
Help to realize the applications such as finer tissue surface reconstruction, the estimation of more accurate camera motion and surgical navigational.
On that basi of the above embodiments, the step S2 specific steps refer to Fig. 2, and Fig. 2 is provided in an embodiment of the present invention
Step S2 specific flow chart in a kind of endoscopic images characteristic point matching method, as shown in Figure 2, comprising:
S21 establishes the unknown deformation between two endoscopic images to be matched and described image characteristic point part neighbour
Corresponding relationship between domain structure；
S22 is based on each described image characteristic point, and corresponding 6 closest approaches construct the office in corresponding point set
Portion's structure；
S23, based between two endoscopic images to be matched unknown deformation and described image characteristic point part it is adjacent
Corresponding relationship between domain structure, and any the distance between the initial matching point pair and its neighborhood point are fixed, and threshold value is arranged
Parameter calculates and obtains the interior point set kept with locality, realizes local distance constraint；
S24, the characteristic point information extracted to the two images to be matched are extended, and addition characteristic point is corresponding
Motion information obtain the characteristic point correspondence set with the Movement consistency using similarity boundary function.
Wherein, the step S22 includes being calculated based on Euclidean distance formula, is obtained based on each described image
Characteristic point and its in corresponding point set corresponding 6 closest approaches come the partial structurtes that construct.
Wherein, similarity boundary function is applied in the step S24, obtains the characteristic point pair with Movement consistency
Set of relationship is answered, is based on obtained by the interior point set kept with locality.
Specifically, step S21 first assumes that unknown deformation has occurred between the image of a pair of of endoscope, and character pair point
Local neighborhood structure can't freely change, can only constrain local neighborhood structure here, and find two width and wait for
The corresponding relationship between unknown deformation and two image characteristic point local neighborhood structures to be matched between matched image, we
It is the optimal solution t for finding locality loss function C by the task presentation for establishing above-mentioned reliable corresponding relationship, with following formula table
Show:
Wherein t is the binary set of N × 1, works as tn=1 and tnCorresponding points (x is respectively indicated when=0n,yn) be interior point or
Outlier, defined parameters ξ is used to the weight of summation first item and Section 2 in balanced double-rope C, and ξ > 0, parameter N are indicated just
The number of beginning matching double points, parameter cnLocal distance structure is calculated by following formula:
Wherein d is the distance matrix for defining two-value, NxAnd NyRespectively indicate the neighborhood point set of characteristic point x and y.
Step S22, the calculation formula based on Euclidean distance, by finding each characteristic point phase in corresponding point set
6 closest approaches answering construct partial structurtes, this is defined herein as k, whenWhen d (xn, xm)=0, whenWhen d
(xn, xm)=1, d (yn, ym) definition similarly.
It is due to characteristics of image point correspondence initial in the step s 21 it has been determined that and any first in step 23
Beginning matching double points and with the distance between its neighborhood point be it is fixed, judged by the way that threshold parameter ζ is arranged, when meeting cn
When≤ξ, tn=1, to solve the optimal solution t of locality loss function C, so that the interior point set that there is locality to keep is obtained,
WithIndicate the interior point set, as follows:
Next in step s 24, by addition motion information and radiology information, by the description information of image characteristic point from
Only location information xn=[xn, yn] expand to xn=[xn, yn；vn；on], wherein n is index a little, vn=[un, vn] indicate
The movement of character pair point in x and y direction, on=[scale, orientation, tilt, rotation] is ASIFT algorithm
The affine information of middle Feature Descriptor.In the set kept with localityOn the basis of, using a similarity boundary functionIt realizes Movement consistency, obtains following formula:
Wherein H () indicates Huber function, for punishing between the function prediction value of estimation and the observation " 1 " of hypothesis
Deviation.G (i, j) is oneSymmetrical gram square matrix, wherein γ be
Standard deviation, λ are the weight of smooth item.It is the Gaussian kernel weight vectors of M dimension,Indicate that M characterization cluster centre exists
Characteristic point { xjWhere space distribution.Therefore, the minimum value in formula is estimated that optimized parameterAs a result, we can be with
Passing through willIt brings intoExpression formula in calculate similarity boundaryValue, then by verifying all initial matchings
As a result SoWhether meet the threshold condition under similarity boundary, is set as εlikelihood, obtain the corresponding relationship with Movement consistency
Set
Step 21 is mentioned to the complete algorithm flow chart of step 24 referring to Fig. 8, Fig. 8 for the embodiment of the present invention in above-described embodiment
The Movement consistency bounding algorithm flow chart kept based on partial structurtes supplied.
The embodiment of the present invention carries out local distance constraint to set by the initial matching point to acquisition, realizes based on part
Structure-preserved Movement consistency constraint, can be from the endoscopic images feature point correspondence there are much noise, accurately
Erroneous matching is rejected on ground, while considering the consistency of invariance and global motion of the partial structurtes in deformation, can be with robust
Ground solves the matching problem under different types of deformation.
On that basi of the above embodiments, the step S3 specific steps refer to Fig. 3, and Fig. 3 is provided in an embodiment of the present invention
Step S3 specific flow chart in a kind of endoscopic images characteristic point matching method, as shown in Figure 3, comprising:
S31 is based on the characteristic point correspondence set, using the affine motion boundary of bilateral variation, obtains described double
Side affine motion consistency model；
The adaptive space threshold parameter of the bilateral moving boundaries is arranged in S32, and it is corresponding interior to obtain the global image
Point matching set, realizes the Feature Points Matching between two endoscopic images to be matched.
Wherein, the adaptive space threshold parameter of the bilateral moving boundaries is set in the step S32, is specifically included:
Model is kept in conjunction with partial structurtes, sets the adaptive space threshold parameter of the bilateral moving boundaries to and institute
The distance restraint threshold parameter for stating partial structurtes is consistent.
Wherein, the corresponding interior point matching set of the global image is calculated and obtained in the step S32, is specifically included:
It is special by the way that estimation result and described image of the described image characteristic point in bilateral moving boundaries both direction is arranged
The distance between sign spot noise observation data threshold value is gathered to obtain the corresponding interior point matching of the global image.Specifically, it is
Model is set to include subtle spatial perception ability, in step S31, in the correspondence set with Movement consistencyThe affine motion boundary of a bilateral variation is cascaded, afterwards to obtain more
Accurate world model, above-mentioned set SlbIt is to obtain in step 24, is expressed as in x and y directionWithIt can estimate to obtain by following optimization problem:
Wherein OkIt is the deviation of scalar, k is the index of different smooth functions.By the loss for minimizing loss function
Value, is calculated optimal solutionWithEach smooth functionIt can be by the way that optimal solution be brought intoExpression formula
In be calculated, while also just obtainingWith
In step 32, in order to make local distance constraint that can play a role in entirely matching process, bilateral moving boundaries
Word space threshold be also required to be consistent therewith.Therefore, keep model in conjunction with partial structurtes, be arranged one it is following adaptive
Capacity-threshold is answered to enable global motion that locality is preferably combined to keep strategy, as follows:
The distance between data threshold value d is observed with noise by the estimation result in setting both directionlp, we can obtain
There is high-precision interior point set .S to followingmc:
Finally realize the Feature Points Matching between two endoscopic images to be matched.
Step 31 is mentioned to the complete algorithm flow chart of step 32 referring to Fig. 9, Fig. 9 for the embodiment of the present invention in above-described embodiment
Supply based on the bilateral moving boundaries bounding algorithm flow chart of adaptive threshold.
The embodiment of the present invention is all had adaptively by the application of adaptive threshold in the Deformation Types in face of different scale
Constraint, be effectively adapted to the endoscopic images of unknown deformation size, can retain in the case where guaranteeing high-precision situation enough
More proper characteristics point correspondence, convenient for being applied in the operation such as subsequent structural remodeling.
Fig. 4 is a kind of endoscopic images Feature Points Matching system structure diagram provided in an embodiment of the present invention, such as Fig. 4 institute
Show, comprising: first processing module 41, Second processing module 42 and third processing module 43, in which: first processing module 41 is used for
Image characteristic point is extracted based on two endoscopic images to be matched, and described image spy is completed based on Feature Descriptor similarity
Sign point matching, obtains the initial matching point of described image characteristic point to set；Second processing module 42 is used to be based on two width
Corresponding relationship between unknown deformation between endoscopic images to be matched and described image characteristic point local neighborhood structure and
Partial structurtes carry out local distance constraint to set to the initial matching point, in conjunction with affine parameter and character pair point movement letter
Breath is extended characteristic point information, estimates similarity boundary, obtains the characteristic point correspondence set with Movement consistency；
Third processing module 43 is used to optimize space length perception based on the characteristic point correspondence set, generates bilateral imitative
Movement consistency model is penetrated, the adaptive distance threshold parameters of bilateral moving boundaries are set, obtains the corresponding interior point of global image
Matching set, realizes the Feature Points Matching between two endoscopic images to be matched.
System provided in an embodiment of the present invention for executing above-mentioned corresponding method, specific embodiment and method
Embodiment is consistent, and the algorithm flow being related to is identical as corresponding method some algorithm process, and details are not described herein again.
The embodiment of the present invention carries out local distance constraint to set by the initial matching point to acquisition, realizes based on part
Structure-preserved Movement consistency constraint, can be from the endoscopic images feature point correspondence there are much noise, accurately
Erroneous matching is rejected on ground, while considering the consistency of invariance and global motion of the partial structurtes in deformation, can be with robust
Ground solves the matching problem under different types of deformation.
On that basi of the above embodiments, Fig. 5 is a kind of endoscopic images Feature Points Matching system provided in an embodiment of the present invention
The Second processing module minor structure schematic diagram of system, as shown in figure 5, Second processing module 42 specifically includes: matched sub-block 421,
Construct submodule 422, the first computational submodule 423 and the second computational submodule 424, in which:
Matched sub-block 421 is used for the unknown deformation and described image established between two endoscopic images to be matched
Corresponding relationship between characteristic point local neighborhood structure；It constructs submodule 422 to be used to be based on each described image characteristic point, right
Corresponding 6 closest approaches are answered in point set to construct the partial structurtes；First computational submodule 423 is used to be based on two width
Corresponding relationship between unknown deformation between endoscopic images to be matched and described image characteristic point local neighborhood structure, and appoint
Anticipating, the distance between the initial matching point pair and its neighborhood point are fixed, and setting threshold parameter is calculated and obtained described with office
The interior point set that portion's property is kept realizes local distance constraint；Second computational submodule 424 is used for the two width figures to be matched
The characteristic point information that picture extracts is extended, and the corresponding motion information of addition characteristic point is obtained using similarity boundary function
To the characteristic point correspondence set with the Movement consistency.
System provided in an embodiment of the present invention for executing above-mentioned corresponding method, specific embodiment and method
Embodiment is consistent, and the algorithm flow being related to is identical as corresponding algorithm process, and details are not described herein again.
The embodiment of the present invention carries out local distance constraint to set by the initial matching point to acquisition, realizes based on part
Structure-preserved Movement consistency constraint, can be from the endoscopic images feature point correspondence there are much noise, accurately
Erroneous matching is rejected on ground, while considering the consistency of invariance and global motion of the partial structurtes in deformation, can be with robust
Ground solves the matching problem under different types of deformation.
On that basi of the above embodiments, Fig. 6 is a kind of endoscopic images Feature Points Matching system provided in an embodiment of the present invention
The third processing module minor structure schematic diagram of system, as shown in fig. 6, third processing module 43 specifically includes: third computational submodule
431 and the 4th computational submodule 432, in which:
Third computational submodule 431 is used to be based on the characteristic point correspondence set, using the affine fortune of bilateral variation
Moving boundary obtains the bilateral affine motion consistency model；4th computational submodule 432 is for being arranged the bilateral movement side
The adaptive space threshold parameter on boundary obtains the corresponding interior point matching set of the global image, and realization described two to be matched
Endoscopic images between Feature Points Matching.
System provided in an embodiment of the present invention for executing above-mentioned corresponding method, specific embodiment and method
Embodiment is consistent, and the algorithm flow being related to is identical as corresponding algorithm process, and details are not described herein again.
The embodiment of the present invention is all had adaptively by the application of adaptive threshold in the Deformation Types in face of different scale
Constraint, be effectively adapted to the endoscopic images of unknown deformation size, can retain in the case where guaranteeing high-precision situation enough
More proper characteristics point correspondence, convenient for being applied in the operation such as subsequent structural remodeling.
Figure 10 illustrates a kind of entity structure schematic diagram of server, and as shown in Figure 10, which may include: processing
Device (processor) 1010, communication interface (Communications Interface) 1020,1030 He of memory (memory)
Communication bus 1040, wherein processor 1010, communication interface 1020, memory 1030 are completed mutually by communication bus 1040
Between communication.Processor 1010 can call the logical order in memory 1030, to execute following method: be waited for based on two width
The endoscopic images matched extract image characteristic point, and complete described image Feature Points Matching based on Feature Descriptor similarity, obtain
Take the initial matching point of described image characteristic point to set；Based on the unknown deformation between two endoscopic images to be matched
Corresponding relationship and partial structurtes between described image characteristic point local neighborhood structure are to the initial matching point to set
Local distance constraint is carried out, characteristic point information is extended in conjunction with affine parameter and character pair point motion information, estimates phase
Like degree boundary, the characteristic point correspondence set with Movement consistency is obtained；Based on the characteristic point correspondence set pair
Space length perception optimizes, and generates bilateral affine motion consistency model, the adaptive distance of bilateral moving boundaries is arranged
Threshold parameter obtains the corresponding interior point matching set of global image, realizes between two endoscopic images to be matched
Feature Points Matching.
In addition, the logical order in above-mentioned memory 1030 can be realized by way of SFU software functional unit and conduct
Independent product when selling or using, can store in a computer readable storage medium.Based on this understanding, originally
Substantially the part of the part that contributes to existing technology or the technical solution can be in other words for the technical solution of invention
The form of software product embodies, which is stored in a storage medium, including some instructions to
So that a computer equipment (can be personal computer, server or the network equipment etc.) executes each implementation of the present invention
The all or part of the steps of example the method.And storage medium above-mentioned include: USB flash disk, mobile hard disk, read-only memory (ROM,
Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic or disk etc. it is various
It can store the medium of program code.
On the other hand, the embodiment of the present invention also provides a kind of non-transient computer readable storage medium, is stored thereon with meter
Calculation machine program, which is implemented to carry out the various embodiments described above offer method when being executed by processor, for example,
Image characteristic point is extracted based on two endoscopic images to be matched, and described image spy is completed based on Feature Descriptor similarity
Sign point matching, obtains the initial matching point of described image characteristic point to set；Based on two endoscopic images to be matched
Between unknown deformation and described image characteristic point local neighborhood structure between corresponding relationship and partial structurtes to described initial
Matching double points set carries out local distance constraint, carries out in conjunction with affine parameter and character pair point motion information to characteristic point information
Extension estimates similarity boundary, obtains the characteristic point correspondence set with Movement consistency；It is corresponding based on the characteristic point
Set of relationship optimizes space length perception, generates bilateral affine motion consistency model, bilateral moving boundaries are arranged
Adaptive distance threshold parameters obtain the corresponding interior point matching set of global image, realize two endoscopes to be matched
Feature Points Matching between image.
The apparatus embodiments described above are merely exemplary, wherein described, unit can as illustrated by the separation member
It is physically separated with being or may not be, component shown as a unit may or may not be physics list
Member, it can it is in one place, or may be distributed over multiple network units.It can be selected according to the actual needs
In some or all of the modules achieve the purpose of the solution of this embodiment.Those of ordinary skill in the art are not paying creativeness
Labour in the case where, it can understand and implement.
Through the above description of the embodiments, those skilled in the art can be understood that each embodiment can
It realizes by means of software and necessary general hardware platform, naturally it is also possible to pass through hardware.Based on this understanding, on
Stating technical solution, substantially the part that contributes to existing technology can be embodied in the form of software products in other words, should
Computer software product may be stored in a computer readable storage medium, such as ROM/RAM, magnetic disk, CD, including several fingers
It enables and using so that a computer equipment (can be personal computer, server or the network equipment etc.) executes each implementation
Method described in certain parts of example or embodiment.
Finally, it should be noted that the above embodiments are merely illustrative of the technical solutions of the present invention, rather than its limitations；Although
Present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that: it still may be used
To modify the technical solutions described in the foregoing embodiments or equivalent replacement of some of the technical features；
And these are modified or replaceed, technical solution of various embodiments of the present invention that it does not separate the essence of the corresponding technical solution spirit and
Range.
Claims (10)
1. a kind of endoscopic images characteristic point matching method characterized by comprising
S1 extracts image characteristic point based on two endoscopic images to be matched, and completes institute based on Feature Descriptor similarity
Image Feature Point Matching is stated, obtains the initial matching point of described image characteristic point to set；
S2, based on the unknown deformation and described image characteristic point local neighborhood structure between two endoscopic images to be matched
Between corresponding relationship and partial structurtes, to the initial matching point to set carry out local distance constraint, and combine it is affine
Parameter and character pair point motion information, are extended characteristic point information, estimate similarity boundary, obtain having movement consistent
The characteristic point correspondence set of property；
S3 optimizes space length perception based on the characteristic point correspondence set, it is consistent to generate bilateral affine motion
Property model, the adaptive distance threshold parameters of bilateral moving boundaries are set, the corresponding interior point matching set of global image is obtained, it is real
Feature Points Matching between existing two endoscopic images to be matched.
2. a kind of endoscopic images characteristic point matching method according to claim 1, which is characterized in that the step of the S2
It specifically includes:
S21 establishes the unknown deformation between two endoscopic images to be matched and described image characteristic point local neighborhood knot
Corresponding relationship between structure；
S22 is based on each described image characteristic point, and corresponding 6 closest approaches are tied in corresponding point set to construct the part
Structure；
S23, based between two endoscopic images to be matched unknown deformation and described image characteristic point local neighborhood knot
Corresponding relationship between structure, and any the distance between the initial matching point pair and its neighborhood point are fixed, and threshold parameter is arranged,
The interior point set kept with locality is calculated and obtained, realizes local distance constraint；
S24, the characteristic point information extracted to the two images to be matched are extended, the corresponding fortune of addition characteristic point
Dynamic information obtains the characteristic point correspondence set with the Movement consistency using similarity boundary function.
3. a kind of endoscopic images characteristic point matching method according to claim 2, which is characterized in that the step S22
Including being calculated based on Euclidean distance formula, obtain based on each described image characteristic point and its in the corresponding point set
Corresponding 6 closest approaches are in conjunction come the partial structurtes that construct.
4. a kind of endoscopic images characteristic point matching method according to claim 3, which is characterized in that the step S24
Middle application similarity boundary function obtains the characteristic point correspondence set with Movement consistency, is based on the tool
Obtained by the interior point set for thering is locality to keep.
5. a kind of endoscopic images characteristic point matching method according to claim 2, which is characterized in that the step of the S3
It specifically includes:
S31 is based on the characteristic point correspondence set, using the affine motion boundary of bilateral variation, obtains described bilateral imitative
Penetrate Movement consistency model；
S32 is arranged the adaptive space threshold parameter of the bilateral moving boundaries, obtains the corresponding interior point of the global image
With set, the Feature Points Matching between two endoscopic images to be matched is realized.
6. a kind of endoscopic images characteristic point matching method according to claim 5, which is characterized in that the step S32
The adaptive space threshold parameter of the middle setting bilateral moving boundaries, specifically includes:
Model is kept in conjunction with partial structurtes, sets the adaptive space threshold parameter of the bilateral moving boundaries to and the office
The distance restraint threshold parameter of portion's structure is consistent.
7. a kind of endoscopic images characteristic point matching method according to claim 6, which is characterized in that the step S32
Middle calculating simultaneously obtains the corresponding interior point matching set of the global image, specifically includes:
Pass through estimation result and described image characteristic point of the setting described image characteristic point in bilateral moving boundaries both direction
The distance between noise observation data threshold value is gathered to obtain the corresponding interior point matching of the global image.
8. a kind of endoscopic images Feature Points Matching system characterized by comprising
First processing module for extracting image characteristic point based on two endoscopic images to be matched, and is described based on feature
Sub- similarity completes described image Feature Points Matching, obtains the initial matching point of described image characteristic point to set；
Second processing module, for based on the unknown deformation and described image feature between two endoscopic images to be matched
Corresponding relationship and partial structurtes between point local neighborhood structure carry out local distance about to set to the initial matching point
Beam is extended characteristic point information in conjunction with affine parameter and character pair point motion information, estimates similarity boundary, is had
There is the characteristic point correspondence set of Movement consistency；
Third processing module is generated double for being optimized based on the characteristic point correspondence set to space length perception
Side affine motion consistency model is arranged the adaptive distance threshold parameters of bilateral moving boundaries, it is corresponding to obtain global image
Interior point matching set, realizes the Feature Points Matching between two endoscopic images to be matched.
9. a kind of electronic equipment including memory, processor and stores the calculating that can be run on a memory and on a processor
Machine program, which is characterized in that the processor is realized as described in any one of claim 1 to 7 when executing described program in one kind
The step of sight glass image characteristic point matching method.
10. a kind of non-transient computer readable storage medium, is stored thereon with computer program, which is characterized in that the computer
Realizing a kind of endoscopic images characteristic point matching method as described in any one of claim 1 to 7 when program is executed by processor
Step.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910521138.9A CN110245671B (en) | 2019-06-17 | 2019-06-17 | Endoscope image feature point matching method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910521138.9A CN110245671B (en) | 2019-06-17 | 2019-06-17 | Endoscope image feature point matching method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110245671A true CN110245671A (en) | 2019-09-17 |
| CN110245671B CN110245671B (en) | 2021-05-28 |
Family
ID=67887435
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910521138.9A Active CN110245671B (en) | 2019-06-17 | 2019-06-17 | Endoscope image feature point matching method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110245671B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111639655A (en) * | 2020-05-20 | 2020-09-08 | 北京百度网讯科技有限公司 | Image local information generation method and device, electronic equipment and storage medium |
| CN112001432A (en) * | 2020-08-12 | 2020-11-27 | 福建农林大学 | Image matching method based on robust feature matching of advanced neighborhood topology consistency |
| CN112784898A (en) * | 2021-01-21 | 2021-05-11 | 大连外国语大学 | Feature point matching method based on local relative motion consistency clustering |
| CN113538295A (en) * | 2021-08-24 | 2021-10-22 | 北京理工大学 | Endoscope weak texture image enhancement method and device |
| CN113538540A (en) * | 2021-08-24 | 2021-10-22 | 北京理工大学 | Medical endoscope continuous frame image feature point matching method and device |
| CN113689555A (en) * | 2021-09-09 | 2021-11-23 | 武汉惟景三维科技有限公司 | Binocular image feature matching method and system |
| CN114078249A (en) * | 2021-11-19 | 2022-02-22 | 武汉大势智慧科技有限公司 | Automatic grouping method and system for front and back overturning images of object |
| CN116385480A (en) * | 2023-02-03 | 2023-07-04 | 腾晖科技建筑智能(深圳)有限公司 | Detection method and system for moving object below tower crane |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101161201A (en) * | 2007-11-08 | 2008-04-16 | 珠海友通科技有限公司 | Method for registrating external circumstance DSA elasticity automatically |
| US20150098659A1 (en) * | 2012-10-26 | 2015-04-09 | Calex Llc | Method and apparatus for image retrieval |
| CN109008909A (en) * | 2018-07-13 | 2018-12-18 | 宜宾学院 | A kind of low-power consumption capsule endoscope Image Acquisition and three-dimensional reconstruction system |
| CN109697692A (en) * | 2018-12-29 | 2019-04-30 | 安徽大学 | One kind being based on the similar feature matching method of partial structurtes |
-
2019
- 2019-06-17 CN CN201910521138.9A patent/CN110245671B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101161201A (en) * | 2007-11-08 | 2008-04-16 | 珠海友通科技有限公司 | Method for registrating external circumstance DSA elasticity automatically |
| US20150098659A1 (en) * | 2012-10-26 | 2015-04-09 | Calex Llc | Method and apparatus for image retrieval |
| CN109008909A (en) * | 2018-07-13 | 2018-12-18 | 宜宾学院 | A kind of low-power consumption capsule endoscope Image Acquisition and three-dimensional reconstruction system |
| CN109697692A (en) * | 2018-12-29 | 2019-04-30 | 安徽大学 | One kind being based on the similar feature matching method of partial structurtes |
Non-Patent Citations (1)
| Title |
|---|
| 郭晓君 等: "基于特征点的内窥镜图像和CT影像配准方法", 《现代商贸工业》 * |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111639655A (en) * | 2020-05-20 | 2020-09-08 | 北京百度网讯科技有限公司 | Image local information generation method and device, electronic equipment and storage medium |
| CN111639655B (en) * | 2020-05-20 | 2023-10-13 | 北京百度网讯科技有限公司 | Image local information generation method, device, electronic equipment and storage medium |
| CN112001432A (en) * | 2020-08-12 | 2020-11-27 | 福建农林大学 | Image matching method based on robust feature matching of advanced neighborhood topology consistency |
| CN112001432B (en) * | 2020-08-12 | 2022-07-08 | 福建农林大学 | Image matching method based on robust feature matching of advanced neighborhood topology consistency |
| CN112784898A (en) * | 2021-01-21 | 2021-05-11 | 大连外国语大学 | Feature point matching method based on local relative motion consistency clustering |
| CN112784898B (en) * | 2021-01-21 | 2024-01-30 | 大连外国语大学 | Feature point matching method based on local relative motion consistency clustering |
| CN113538295A (en) * | 2021-08-24 | 2021-10-22 | 北京理工大学 | Endoscope weak texture image enhancement method and device |
| CN113538540A (en) * | 2021-08-24 | 2021-10-22 | 北京理工大学 | Medical endoscope continuous frame image feature point matching method and device |
| CN113538540B (en) * | 2021-08-24 | 2024-08-06 | 北京理工大学 | Method and device for matching feature points of continuous frame images of medical endoscope |
| CN113689555B (en) * | 2021-09-09 | 2023-08-22 | 武汉惟景三维科技有限公司 | Binocular image feature matching method and system |
| CN113689555A (en) * | 2021-09-09 | 2021-11-23 | 武汉惟景三维科技有限公司 | Binocular image feature matching method and system |
| CN114078249A (en) * | 2021-11-19 | 2022-02-22 | 武汉大势智慧科技有限公司 | Automatic grouping method and system for front and back overturning images of object |
| CN114078249B (en) * | 2021-11-19 | 2024-08-06 | 武汉大势智慧科技有限公司 | Automatic grouping method and system for object front and back face overturning images |
| CN116385480A (en) * | 2023-02-03 | 2023-07-04 | 腾晖科技建筑智能(深圳)有限公司 | Detection method and system for moving object below tower crane |
| CN116385480B (en) * | 2023-02-03 | 2023-10-20 | 腾晖科技建筑智能(深圳)有限公司 | Detection method and system for moving object below tower crane |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110245671B (en) | 2021-05-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110245671A (en) | A kind of endoscopic images characteristic point matching method and system | |
| García-Peraza-Herrera et al. | Real-time segmentation of non-rigid surgical tools based on deep learning and tracking | |
| Lin et al. | Video‐based 3D reconstruction, laparoscope localization and deformation recovery for abdominal minimally invasive surgery: a survey | |
| CN107358217B (en) | Sight estimation method and device | |
| KR101532864B1 (en) | Planar mapping and tracking for mobile devices | |
| WO2016119117A1 (en) | Localization and mapping method | |
| CN109697728A (en) | Data processing method, device, system and storage medium | |
| CN109357633B (en) | Three-dimensional scanning method, device, storage medium and processor | |
| WO2015154205A1 (en) | Methods and systems for verifying face images based on canonical images | |
| CN113689503B (en) | Target object posture detection method, device, equipment and storage medium | |
| US20210397254A1 (en) | Eye tracking in near-eye displays | |
| CN109145783B (en) | Method and apparatus for generating information | |
| US10937192B2 (en) | Resolving incorrect distributed simultaneous localization and mapping (SLAM) data in edge cloud architectures | |
| JP2014032623A (en) | Image processor | |
| CN115035004A (en) | Image processing method, apparatus, device, readable storage medium and program product | |
| CN114724148A (en) | Method, apparatus, device, medium and product for generating model and extracting feature | |
| CN110533775A (en) | A kind of glasses matching process, device and terminal based on 3D face | |
| CN114120433B (en) | Image processing method, image processing apparatus, electronic device, and medium | |
| EP3836073B1 (en) | Method and apparatus for tracking eye based on eye reconstruction | |
| Watanabe et al. | A new 2D depth-depth matching algorithm whose translation and rotation freedoms are separated | |
| CN114723973A (en) | Image feature matching method and device for large-scale change robustness | |
| Picos et al. | Evolutionary correlation filtering based on pseudo-bacterial genetic algorithm for pose estimation of highly occluded targets | |
| TWI853460B (en) | Method for determining two-eye gaze point and host | |
| Lin | Visual SLAM and Surface Reconstruction for Abdominal Minimally Invasive Surgery | |
| JP7528383B2 (en) | System and method for training a model for predicting dense correspondences in images using geodesic distances - Patents.com |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |