CN112927276B - Image registration method, device, electronic equipment and storage medium - Google Patents
Image registration method, device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN112927276B CN112927276B CN202110260786.0A CN202110260786A CN112927276B CN 112927276 B CN112927276 B CN 112927276B CN 202110260786 A CN202110260786 A CN 202110260786A CN 112927276 B CN112927276 B CN 112927276B
- Authority
- CN
- China
- Prior art keywords
- image
- grid
- registered
- feature point
- grid region
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 60
- 238000013507 mapping Methods 0.000 claims abstract description 47
- 239000011159 matrix material Substances 0.000 claims description 163
- 238000004590 computer program Methods 0.000 claims description 12
- 238000009499 grossing Methods 0.000 claims description 9
- 238000000354 decomposition reaction Methods 0.000 claims description 8
- 238000001514 detection method Methods 0.000 claims description 7
- 239000013598 vector Substances 0.000 description 11
- 230000008569 process Effects 0.000 description 10
- 238000004891 communication Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 6
- 238000012545 processing Methods 0.000 description 5
- 230000009466 transformation Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000008094 contradictory effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/32—Indexing scheme for image data processing or generation, in general involving image mosaicing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30236—Traffic on road, railway or crossing
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the application provides an image registration method, an image registration device, electronic equipment and a storage medium, wherein an image to be registered is divided into a plurality of grid areas, local homography matrixes of each grid area are calculated respectively, and the local homography matrixes are used for mapping the corresponding grid areas respectively.
Description
Technical Field
The present disclosure relates to the field of image processing technologies, and in particular, to an image registration method, an image registration device, an electronic device, and a storage medium.
Background
Image registration refers to geometrically calibrating a reference image and an image to be registered for a process of registering two images with overlapping areas taken of the same scene with the same or different image acquisition devices. The difference between the two images is mainly from different imaging conditions, image registration is also called image matching. The mathematical description of image registration may be defined as a spatial transformation between the images to be registered. It is required that a part of the targets between the image to be registered and the calibration reference image are identical.
In the existing image registration method, a group of key points representing the same target and the same point in a calibration reference image and an image to be registered are matched, a homography matrix from the image to be registered to the calibration reference image is calculated by utilizing a plurality of groups of key points, and then the whole image to be registered is registered by utilizing the homography matrix.
However, in the above method for image registration, the image to be registered is registered by using the homography matrix of the whole image to be registered, which requires that the object in the image to be registered is a planar object or a non-planar object but the angle of view only rotates, otherwise, the double image problem occurs in the registered image.
Disclosure of Invention
An embodiment of the application aims to provide an image registration method, an image registration device, electronic equipment and a storage medium, so as to reduce the ghost problem of registered images. The specific technical scheme is as follows:
in a first aspect, an embodiment of the present application provides an image registration method, including:
acquiring an image to be registered and a reference image, wherein the image to be registered and the reference image comprise the same target;
performing feature point detection and feature point matching on the image to be registered and the reference image to obtain a first feature point set of the image to be registered, a second feature point set of the reference image, and matching relations between each first feature point in the first feature point set and each second feature point in the second feature point set, wherein the first feature point and the second feature point with the matching relations represent the same point in the same target;
Dividing the image to be registered into a plurality of grid areas;
calculating a local homography matrix of each grid region in the image to be registered according to the first feature point set, the second feature point set and the matching relation;
for each grid region in the image to be registered, mapping pixels in the grid region by utilizing a local homography matrix of the grid region to obtain a mapping region of the grid region;
and splicing the mapping areas of the grid areas to obtain the target image after registration of the images to be registered.
In a possible embodiment, after the dividing the image to be registered into a plurality of grid areas, the method comprises:
for each grid region of the image to be registered, respectively calculating the weight of each first characteristic point relative to the grid region so as to obtain a weight matrix of the grid region, wherein the weight matrix of the grid region consists of the weight of each first characteristic point relative to the grid region, and for any first characteristic point, the weight of the first characteristic point relative to the grid region is inversely related to the distance between the first characteristic point and the key point of the grid region;
The calculating a local homography matrix of each grid region in the image to be registered according to the first feature point set, the second feature point set and the matching relation comprises the following steps:
and calculating a local homography matrix of each grid region in the image to be registered according to the weight matrix of the grid region, the first characteristic point set, the second characteristic point set and the matching relation.
In one possible embodiment, the method further comprises:
for each grid region, the center points of all pixels in the grid region are used as key points of the grid region.
In one possible embodiment, the method further comprises:
determining a foreground target area in each grid area;
for any grid area, taking the central points of all pixels in the foreground target area of the grid area as key points of the grid area under the condition that the grid area comprises the foreground target area;
for any grid region, in the case that the grid region does not include a foreground target region, the center points of all pixels in the grid region are taken as key points of the grid region.
In a possible implementation manner, the stitching the mapping areas of each grid area to obtain the target image after the registration of the image to be registered includes:
and splicing the mapping areas of the grid areas, and smoothing pixels at the spliced positions to obtain the target image after registration of the images to be registered.
In one possible implementation manner, the calculating weights of the first feature points relative to the grid area for each grid area of the image to be registered, so as to obtain a weight matrix of the grid area includes:
for each grid region of the image to be registered, determining the weight of each first feature point relative to the grid region based on the distance between each first feature point and the key point of the grid region;
and determining a weight matrix of each grid region of the image to be registered according to the weight of each first characteristic point relative to the grid region.
In a possible implementation manner, for each grid region in the image to be registered, the calculating a local homography matrix of the grid region according to the weight matrix of the grid region, the first feature point set, the second feature point set, and the matching relationship includes:
Determining a feature point matrix according to the first feature point set, the second feature point set and the matching relation;
for each grid region in the image to be registered, multiplying the weight matrix of the grid region with the characteristic point matrix to obtain a corrected matrix;
and carrying out singular value decomposition on the corrected matrix to obtain a local homography matrix of the grid region.
In a second aspect, embodiments of the present application provide an image registration apparatus, the apparatus including:
the image acquisition module is used for acquiring an image to be registered and a reference image, wherein the image to be registered and the reference image comprise the same target;
the feature point acquisition module is used for carrying out feature point detection and feature point matching on the image to be registered and the reference image to obtain a first feature point set of the image to be registered, a second feature point set of the reference image, and a matching relation between each first feature point in the first feature point set and each second feature point in the second feature point set, wherein the first feature point and the second feature point with the matching relation represent the same point in the same target;
The grid region dividing module is used for dividing the image to be registered into a plurality of grid regions;
the local homography matrix determining module is used for calculating a local homography matrix of each grid area in the image to be registered according to the first characteristic point set, the second characteristic point set and the matching relation;
the grid region mapping module is used for mapping pixels in each grid region in the image to be registered by utilizing a local homography matrix of the grid region to obtain a mapping region of the grid region;
and the mapping region splicing module is used for splicing the mapping regions of the grid regions to obtain the target image after the registration of the images to be registered.
In one possible embodiment, the apparatus further comprises:
the weight matrix determining module is used for respectively calculating the weight of each first characteristic point relative to each grid area of the image to be registered so as to obtain a weight matrix of the grid area, wherein the weight matrix of the grid area consists of the weight of each first characteristic point relative to the grid area, and the weight of the first characteristic point relative to the grid area and the distance between the first characteristic point and the key point of the grid area are inversely related to any first characteristic point;
The local homography matrix determining module is specifically configured to: and calculating a local homography matrix of each grid region in the image to be registered according to the weight matrix of the grid region, the first characteristic point set, the second characteristic point set and the matching relation.
In one possible embodiment, the apparatus further comprises:
and the key point determining module is used for taking the central points of all pixels in each grid area as key points of the grid area.
In one possible embodiment, the apparatus further comprises:
the key point determining module is used for determining foreground target areas in all grid areas; for any grid area, taking the central points of all pixels in the foreground target area of the grid area as key points of the grid area under the condition that the grid area comprises the foreground target area; for any grid region, in the case that the grid region does not include a foreground target region, the center points of all pixels in the grid region are taken as key points of the grid region.
In a possible implementation manner, the mapping area stitching module is specifically configured to: and splicing the mapping areas of the grid areas, and smoothing pixels at the spliced positions to obtain the target image after registration of the images to be registered.
In a possible implementation manner, the weight matrix determining module is specifically configured to: for each grid region of the image to be registered, determining the weight of each first feature point relative to the grid region based on the distance between each first feature point and the key point of the grid region; and determining a weight matrix of each grid region of the image to be registered according to the weight of each first characteristic point relative to the grid region.
In one possible implementation manner, the local homography matrix determining module is specifically configured to: determining a feature point matrix according to the first feature point set, the second feature point set and the matching relation; for each grid region in the image to be registered, multiplying the weight matrix of the grid region with the characteristic point matrix to obtain a corrected matrix; and carrying out singular value decomposition on the corrected matrix to obtain a local homography matrix of the grid region.
In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory;
the memory is used for storing a computer program;
the processor is configured to implement any one of the image registration methods described in the present application when executing the program stored in the memory.
In a fourth aspect, embodiments of the present application provide a computer readable storage medium having a computer program stored therein, which when executed by a processor, implements the image registration method of any of the present application.
The beneficial effects of the embodiment of the application are that:
according to the image registration method, the device, the electronic equipment and the storage medium, the image to be registered is divided into a plurality of grid areas, the local homography matrix of each grid area is calculated respectively, the local homography matrix is used for mapping the corresponding grid area respectively, and compared with the method that the homography matrix of the whole image to be registered is used for directly mapping the whole image to be registered, the local homography matrix of each grid area can be different, even if targets in different grid areas are not plane objects or motion other than rotational motion occurs, the local homography matrix of each grid area can be used for mapping, so that the ghost problem of the registered image can be reduced. Of course, not all of the above-described advantages need be achieved simultaneously in practicing any one of the products or methods of the present application.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1a is a first schematic illustration of an image registration method according to an embodiment of the present application;
FIG. 1b is a second schematic illustration of an image registration method according to an embodiment of the present application;
fig. 2a is a first schematic diagram of mesh region division in an image to be registered according to an embodiment of the present application;
FIG. 2b is a second schematic diagram of a target image composed of mapping regions according to an embodiment of the present application;
FIG. 3 is a third schematic illustration of an image registration method according to an embodiment of the present application;
FIG. 4a is a schematic illustration of a single target image to be registered according to an embodiment of the present application;
FIG. 4b is a schematic illustration of a single target reference image according to an embodiment of the present application;
FIG. 4c is a schematic diagram of the related art in which the entire single-target image to be registered is directly registered by using the homography matrix of the entire single-target image to be registered;
FIG. 4d is a schematic illustration of a single target image registered using the image registration method of the embodiments of the present application;
FIG. 5a is a schematic illustration of a multi-target image to be registered according to an embodiment of the present application;
FIG. 5b is a schematic diagram of a multi-target reference image according to an embodiment of the present application;
FIG. 5c is a schematic diagram of the related art in which the entire multi-target image to be registered is directly registered using the homography matrix of the entire multi-target image to be registered;
FIG. 5d is a schematic illustration of a multi-target image registered using the image registration method of the embodiments of the present application;
fig. 6 is a first schematic view of an image registration apparatus according to an embodiment of the present application;
fig. 7 is a schematic diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application. Based on the embodiments herein, a person of ordinary skill in the art would be able to obtain all other embodiments based on the disclosure herein, which are within the scope of the disclosure herein.
First, the terms of art in this application are explained:
Homogeneous coordinates (Homogeneous coordinates): the homogeneous coordinates are obtained by expressing two-dimensional coordinates (x, y) in three dimensions (x, y, w) ∈P 3 And three-dimensional coordinates (x, y, z) ∈R 3 In contrast, homogeneous coordinates have only two degrees of freedom.
Where w represents the scale, when w=1,
image registration: and matching or superposing two or more images acquired at different times, with different equipment or under different conditions (pose, climate, angle and the like).
Homography matrix (Homography): homography is a concept of projection geometry that maps points (three-dimensional homogeneous vectors) on one projection plane to another projection plane, maps straight lines to straight lines, and has a straight line preserving property. In general, homography is a linear transformation of three-dimensional homogeneous vectors, which can be represented by a 3 x 3 non-singular matrix H,
this is an equation for homogeneous coordinates, H times a non-zero scale factor, which is still true, i.e., H is a 3 x 3 homogeneous matrix with 8 unknowns.
Local homography matrix: dividing the image into C 1 ×C 2 Each unit corresponds to one officeAnd (5) a part homography matrix.
The DLT (Direct Linear Transform, direct linear transformation) algorithm is a common algorithm for calculating a homography matrix, and specifically, the process of calculating the homography matrix by the DLT algorithm includes:
Characteristic points successfully matched in the image I to be matched and the reference image I' are respectively expressed as X= [ X ] i ,y i ] T And X' = [ X i ',y i '] T I= (1, …, N), where x i Is the abscissa, y, of the ith feature point in I i Is the ordinate, x of the ith feature point in I i ' is the abscissa, y, of the ith feature point in I i The 'is the ordinate of the ith feature point in the I', the ith feature point in the I 'is matched with the ith feature point in the I', and N is the number of feature points in the image to be matched. The projection matrices for I and I' are shown below:
and->The homogeneous coordinates of the matched feature points X and X', respectively. H is a homography matrix of I to I':
expressed in terms of vectors: h= [ H ] 1 ,h 2 ,h 3 ] T 。
X＝[x i ,y i ] T Conversion to X' = [ X i ',y i '] T The process of i= (1, …, N) is as follows:
the vector representation method of the h matrix is as follows:
h＝(h 11 h 12 h 13 h 21 h 22 h 23 h 31 h 32 h 33 ) T (5)
(3) And (4) the effect after deployment is as follows:
-h 11 x i -h 12 y i -h 13 +(h 31 x i +h 32 y i +h 33 )x i '＝0 (6)
-h 21 x i -h 22 y i -h 23 +(h 31 x i +h 32 y i +h 33 )y i '＝0 (7)
equations (6) and (7) can be expressed as the product of the feature point matrix a and the vector h as follows:
wherein A is E R 2N×9 I.e. a matrix of 2N rows and 9 columns, N being the number of matched feature points.
As can be seen from (8), when the number of matched feature points is 1, i.e. when i=1, there are two equation sets, and the vector h has 9 unknowns, so at least 5 matched feature points are needed to construct 10 equation sets to solve the h vector.
The solving method of h is to perform singular value decomposition on A to obtain A=U ΣV T Where the right singular vector V is the h vector. After the vector H is obtained, the vector H is deformed to obtain a matrix H, and any point in I is subjected toMappingTo the point corresponding to I->The transformation relationship of (2) is as follows:
so that registration of I can be achieved.
As can be seen from the above process, the homography matrix is suitable for registration of a planar scene or a scene in which only a target rotates, and when the environment in the registered scene is complex, for example, when the registered target includes a plurality of curved surfaces, a ghost phenomenon occurs when the whole image is mapped by using the homography matrix. This is because the homography matrix is a mapping matrix between two planes, but feature points that match successfully do not necessarily satisfy on the same plane. Therefore, the DLT algorithm calculates the homography matrix by fitting a matrix H so that ah=0 can satisfy most of the feature points, which are called inner points, and the rest are outer points, and the image ghost of the outer points and the surrounding of the outer points after registration is often serious, where a is a feature point matrix determined according to the feature points.
In order to reduce the ghost problem of the registered image, the embodiment of the present application provides an image registration method, referring to fig. 1a, the method includes:
S101, acquiring an image to be registered and a reference image, wherein the image to be registered and the reference image comprise the same target.
The image registration method of the embodiment of the application can be implemented by electronic equipment, and specifically, the electronic equipment can be equipment such as an intelligent video camera, a hard disk video recorder, a server, a personal computer or a smart phone.
The image to be registered and the reference image may be two images captured by the same or different image capturing devices, and the image to be registered and the reference image include the same target, for example, the image to be registered and the reference image include the same vehicle, pedestrian, building, or the like.
And S102, carrying out feature point detection and feature point matching on the image to be registered and the reference image to obtain a first feature point set of the image to be registered, a second feature point set of the reference image, and a matching relation between each first feature point in the first feature point set and each second feature point in the second feature point set, wherein the first feature point and the second feature point with the matching relation represent the same point in the same target.
The feature points in the image to be registered and the reference image can be detected and matched through a related feature point detection algorithm, for example, a SIFT (Scale-invariant feature transform, scale invariant feature transform) algorithm, a SURF (Speeded Up Robust Features) algorithm, a ORB (Oriented Fast and Rotated Brief) algorithm and the like, wherein the feature points in the image to be registered are called first feature points, the feature points in the reference image are called second feature points, the number of the first feature points is the same as the number of the second feature points, and the first feature points are matched with the second feature points one by one.
And S103, dividing the image to be registered into a plurality of grid areas.
The grid region dividing method of the image to be registered may be set in a self-defined manner according to the actual situation, in one example, the image to be registered may be divided into a preset grid number of grid regions according to the preset grid number, for example, the preset grid number is C1×c2, the width of each grid region is obtained by dividing the width of the image to be registered by C1, the height of each grid region is obtained by dividing the height of the image to be registered by C2, and then the image to be registered is divided into C1×c2 rectangular grid regions according to the width and the height of each grid region.
In one example, the image to be registered may be divided into a plurality of grid areas according to the positions of the first feature points, wherein each grid area includes at least 5 first feature points therein. In this embodiment, each grid region includes at least 5 first feature points, and the local homography matrix of the grid region can be directly calculated according to the first feature points in the grid region and the second feature points in the corresponding reference image.
And S104, calculating a local homography matrix of each grid region in the image to be registered according to the first characteristic point set, the second characteristic point set and the matching relation.
For each grid region, according to the first feature point set, the second feature point set and the matching relation, calculating a homography matrix of the grid region by using a DLT algorithm to serve as a local homography matrix of the grid region.
S105, for each grid region in the image to be registered, mapping pixels in the grid region by utilizing a local homography matrix of the grid region to obtain a mapping region of the grid region.
And for each grid region, mapping each pixel in the grid region by utilizing a local homography matrix of the grid region to obtain a mapping region of the grid region.
And S106, splicing the mapping areas of the grid areas, and processing to obtain the target image after registration of the images to be registered.
And splicing the mapping areas of the grid areas according to the positions of the grid areas to obtain spliced images.
In order to further increase the display effect of the target image, in a possible implementation manner, referring to fig. 1b, the stitching the mapping areas of each grid area, processing to obtain the target image after the registration of the image to be registered includes: s1061, stitching the mapping areas of the grid areas, and smoothing the pixels at the stitching points to obtain the target image after the registration of the images to be registered.
And smoothing pixels at the joint of each mapping area aiming at the spliced image, so as to obtain a target image after registration of the images to be registered. In one example, the pixel to be smoothed may be determined according to the width and the height of the grid region, for example, a% of the width of the grid region is selected as the pixel width to be smoothed in the vertical axis direction, and B% of the height of the grid region is selected as the pixel width to be smoothed in the horizontal axis direction, where a and B mean values are preset constants; the pixel width in the vertical axis direction and the horizontal axis direction, which needs to be smoothed, may be set in advance. In an example, taking the width of the pixel that needs to be smoothed in the vertical axis direction as C and the width of the pixel that needs to be smoothed in the horizontal axis direction as D as an example, the pixel at the joint that needs to be smoothed may be as shown in fig. 2a and fig. 2b, where fig. 2a is a schematic view of the pixel at the joint that needs to be smoothed in the vertical axis direction of the two mapping regions, and fig. 2b is a schematic view of the pixel at the joint that needs to be smoothed in the horizontal axis direction of the two mapping regions. The specific manner of the pixel smoothing process may be referred to as a manner of the pixel smoothing process in the related art, and is not specifically limited in this application.
In the embodiment of the application, the image to be registered is divided into a plurality of grid areas, the local homography matrix of each grid area is calculated respectively, and the local homography matrix is used for mapping the corresponding grid area respectively.
In a possible embodiment, referring to fig. 3, after the dividing the image to be registered into a plurality of grid areas, the method includes:
s107, for each grid region of the image to be registered, calculating the weight of each first feature point relative to the grid region, so as to obtain a weight matrix of the grid region, wherein the weight matrix of the grid region consists of the weights of each first feature point relative to the grid region, and for any first feature point, the weights of the first feature points relative to the grid region are inversely related to the distance between the first feature points relative to key points of the grid region.
The weight of the first feature point relative to the grid region is related to the distance of the first feature point relative to the key point of the grid region, where the distance includes, but is not limited to, a gaussian distance, a euclidean distance, a straight line distance, and the like. In one example, the greater the distance of a first feature point relative to a key point of the grid region, the less the weight of the first feature point relative to the grid region. The key points of the grid area can be selected in a self-defined manner according to actual conditions, for example, the center points of all pixels in the grid area can be selected as the key points of the grid area; or the central points of all the first characteristic points in the grid area can be selected as key points of the grid area; or the center points of all pixels in the foreground object area of the grid area can be selected as key points of the grid area, etc.
In one possible embodiment, the method further comprises: for each grid region, the center points of all pixels in the grid region are used as key points of the grid region.
In one possible embodiment, the method further comprises:
step one, a foreground target area in each grid area is determined.
The foreground object regions in each grid region may be determined using computer vision techniques.
And step two, regarding any grid area, taking the central points of all pixels in the foreground target area of the grid area as key points of the grid area when the grid area comprises the foreground target area.
And thirdly, regarding any grid area, taking the central points of all pixels in the grid area as key points of the grid area under the condition that the grid area does not comprise a foreground target area.
The calculating a local homography matrix of each grid region in the image to be registered according to the first feature point set, the second feature point set and the matching relation includes:
s1041, for each grid region in the image to be registered, calculating a local homography matrix of the grid region according to the weight matrix of the grid region, the first feature point set, the second feature point set and the matching relation.
In the process of calculating the local homography matrix of the grid region, the weight matrix may be used to calculate the weight of each feature point, so as to obtain a local homography matrix related to the weight, for example, when the homography matrix of the grid region is calculated by using the DLT algorithm, the weight matrix of the grid region may be multiplied left, so as to obtain the homography matrix of the grid region as the local homography matrix of the grid region.
In one possible implementation manner, the calculating weights of the first feature points with respect to the grid area for each grid area of the image to be registered, so as to obtain a weight matrix of the grid area includes:
and step A, determining the weight of each first characteristic point relative to each grid area of the image to be registered based on the distance between each first characteristic point and the key point of the grid area.
The distances herein include, but are not limited to, gaussian distances, euclidean distances, straight line distances, and the like. In one example, taking a gaussian distance as an example, for each grid region of the image to be registered, the weight of each first feature point relative to the grid region is calculated according to the following formula:
For each grid region of the image to be registered, respectively calculating Gaussian distance weights of key points of the first feature points relative to the grid region as weights of the first feature points relative to the grid region:
wherein,x is the weight of the ith first feature point relative to the grid area * X is the coordinates of the key points of the grid region i For the coordinates of the i-th first feature point, σ is a constant set in advance, exp () represents an exponential function based on a natural constant e.
Sigma is a preset constant, and can be set in a self-defined manner according to practical situations, and in one example, the value range of sigma is (0,0.9).
And B, determining a weight matrix of each grid region of the image to be registered according to the weight of each first characteristic point relative to the grid region.
And combining the weights of the first characteristic points relative to each grid region of the image to be registered into a weight matrix of the grid region. In one example, for each grid region of the image to be registered, a weight matrix of the grid region is obtained according to weights of the first feature points relative to the grid region:
Wherein W is * For the weight matrix of the grid region, diag represents a diagonal matrix, and a is a preset constant. In one example, a may be set to 1 or 0. Weight matrix W * Is a diagonal matrix, and the diagonal matrix has 2N elements, each weight valueTwo consecutive positions are taken up on the diagonal.
In one possible implementation manner, the calculating, for each grid region in the image to be registered, a local homography matrix of the grid region according to a weight matrix of the grid region, the first feature point set, the second feature point set, and the matching relationship includes:
and step 1, determining a feature point matrix according to the first feature point set, the second feature point set and the matching relation.
In one example, the feature point matrix a may be:
wherein A epsilonR 2N×9 I.e. a matrix of 2N rows and 9 columns, N being the number of first feature points/second feature points, x i Is the abscissa of the ith first feature point, y i Is the ordinate, x of the ith first feature point i ' is the abscissa of the ith second feature point, y i ' is the ordinate of the ith second feature point;
step 2, for each grid region in the image to be registered, multiplying the weight matrix of the grid region with the characteristic point matrix to obtain a corrected matrix; and carrying out singular value decomposition on the corrected matrix to obtain a local homography matrix of the grid region.
For each grid region in the image to be registered, multiplying the weight matrix of the grid region with the characteristic point matrix A to obtain a corrected matrix A * For A * And carrying out singular value decomposition to obtain a local homography matrix of the grid region.
In one example, an image to be registered is shown in fig. 4a, a reference image is shown in fig. 4b, the whole image to be registered is directly registered by using a homography matrix of the whole image to be registered, the registered image is shown in fig. 4c, and the registered image by using the image registration method in the embodiment of the present application is shown in fig. 4 d. Therefore, the ghost effect of the images registered by the image registration method in the embodiment of the application at the license plate, the vehicle window and the logo is obviously reduced, and the effect is obviously optimized.
In one example, an image to be registered is shown in fig. 5a, a reference image is shown in fig. 5b, the whole image to be registered is directly registered by using a homography matrix of the whole image to be registered, the registered image is shown in fig. 5c, and the registered image by using the image registration method in the embodiment of the present application is shown in fig. 5 d. Therefore, the ghost effect of the images registered by the image registration method in the embodiment of the application at the license plate, the vehicle window and the logo is obviously reduced, and the effect is obviously optimized.
The embodiment of the application also provides an image registration device, referring to fig. 6, the device includes:
an image acquisition module 11, configured to acquire an image to be registered and a reference image, where the image to be registered and the reference image include the same target;
the feature point obtaining module 12 is configured to perform feature point detection and feature point matching on the image to be registered and the reference image to obtain a first feature point set of the image to be registered, a second feature point set of the reference image, and a matching relationship between each first feature point in the first feature point set and each second feature point in the second feature point set, where the first feature point and the second feature point having the matching relationship represent the same point in the same target;
a grid region dividing module 13, configured to divide the image to be registered into a plurality of grid regions;
the local homography matrix determining module 14 is configured to calculate a local homography matrix of each grid region in the image to be registered according to the first feature point set, the second feature point set, and the matching relationship;
a grid region mapping module 15, configured to map, for each grid region in the image to be registered, pixels in the grid region by using a local homography matrix of the grid region, so as to obtain a mapping region of the grid region;
And the mapping region stitching module 16 is configured to stitch the mapping regions of each grid region to obtain the target image after the registration of the images to be registered.
In one possible embodiment, the apparatus further comprises:
the weight matrix determining module is used for respectively calculating the weight of each first characteristic point relative to each grid area of the image to be registered so as to obtain a weight matrix of the grid area, wherein the weight matrix of the grid area consists of the weight of each first characteristic point relative to the grid area, and the weight of the first characteristic point relative to the grid area and the distance between the first characteristic point and the key point of the grid area are inversely related to any first characteristic point;
the local homography matrix determining module is specifically configured to: and calculating a local homography matrix of each grid region in the image to be registered according to the weight matrix of the grid region, the first characteristic point set, the second characteristic point set and the matching relation.
In one possible embodiment, the apparatus further comprises:
and the key point determining module is used for taking the central points of all pixels in each grid area as key points of the grid area.
In one possible embodiment, the apparatus further comprises:
the key point determining module is used for determining foreground target areas in all grid areas; for any grid area, taking the central points of all pixels in the foreground target area of the grid area as key points of the grid area under the condition that the grid area comprises the foreground target area; for any grid region, in the case that the grid region does not include a foreground target region, the center points of all pixels in the grid region are taken as key points of the grid region.
In a possible implementation manner, the mapping area stitching module is specifically configured to: and splicing the mapping areas of the grid areas, and smoothing pixels at the spliced positions to obtain the target image after registration of the images to be registered.
In a possible implementation manner, the weight matrix determining module is specifically configured to: for each grid region of the image to be registered, determining the weight of each first feature point relative to the grid region based on the distance between each first feature point and the key point of the grid region; and determining a weight matrix of each grid region of the image to be registered according to the weight of each first characteristic point relative to the grid region.
In one possible implementation manner, the local homography matrix determining module is specifically configured to: determining a feature point matrix according to the first feature point set, the second feature point set and the matching relation; for each grid region in the image to be registered, multiplying the weight matrix of the grid region with the characteristic point matrix to obtain a corrected matrix; and carrying out singular value decomposition on the corrected matrix to obtain a local homography matrix of the grid region.
The embodiment of the application also provides electronic equipment, which comprises: a processor and a memory;
the memory is used for storing a computer program;
the processor is configured to implement any one of the image registration methods described in the present application when executing the computer program stored in the memory.
Optionally, referring to fig. 7, in addition to the processor 21 and the memory 23, the electronic device in the embodiment of the present application further includes a communication interface 22 and a communication bus 24, where the processor 21, the communication interface 22, and the memory 23 complete communication with each other through the communication bus 24.
The communication bus mentioned for the above-mentioned electronic devices may be a PCI (Peripheral Component Interconnect, peripheral component interconnect standard) bus or an EISA (Extended Industry Standard Architecture ) bus, or the like. The communication bus may be classified as an address bus, a data bus, a control bus, or the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.
The communication interface is used for communication between the electronic device and other devices.
The Memory may include RAM (Random Access Memory ) or NVM (Non-Volatile Memory), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the aforementioned processor.
The processor may be a general-purpose processor, including a CPU (Central Processing Unit ), NP (Network Processor, network processor), etc.; but also DSP (Digital Signal Processing, digital signal processor), ASIC (Application Specific Integrated Circuit ), FPGA (Field-Programmable Gate Array, field programmable gate array) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components.
The embodiment of the application also provides a computer readable storage medium, and a computer program is stored in the computer readable storage medium, and when the computer program is executed by a processor, the image registration method of any one of the application is realized.
In yet another embodiment provided herein, there is also provided a computer program product containing instructions that, when run on a computer, cause the computer to perform the image registration method of any of the present applications.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital subscriber line), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), etc.
It should be noted that, in this document, the technical features in each alternative may be combined to form a solution, so long as they are not contradictory, and all such solutions are within the scope of the disclosure of the present application. Relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for embodiments of the apparatus, electronic device, computer program product and storage medium, the description is relatively simple, as it is substantially similar to the method embodiments, as relevant see also part of the description of the method embodiments.
The foregoing description is only of the preferred embodiments of the present application and is not intended to limit the scope of the present application. Any modifications, equivalent substitutions, improvements, etc. that are within the spirit and principles of the present application are intended to be included within the scope of the present application.
Claims (8)
1. A method of image registration, the method comprising:
acquiring an image to be registered and a reference image, wherein the image to be registered and the reference image comprise the same target;
performing feature point detection and feature point matching on the image to be registered and the reference image to obtain a first feature point set of the image to be registered, a second feature point set of the reference image, and matching relations between each first feature point in the first feature point set and each second feature point in the second feature point set, wherein the first feature point and the second feature point with the matching relations represent the same point in the same target;
dividing the image to be registered into a plurality of grid areas;
calculating a local homography matrix of each grid region in the image to be registered according to the first feature point set, the second feature point set and the matching relation;
For each grid region in the image to be registered, mapping pixels in the grid region by utilizing a local homography matrix of the grid region to obtain a mapping region of the grid region;
splicing the mapping areas of each grid area to obtain a target image after registration of the images to be registered;
after the dividing the image to be registered into a plurality of grid areas, the method comprises:
for each grid region of the image to be registered, respectively calculating the weight of each first characteristic point relative to the grid region so as to obtain a weight matrix of the grid region, wherein the weight matrix of the grid region consists of the weight of each first characteristic point relative to the grid region, and for any first characteristic point, the weight of the first characteristic point relative to the grid region is inversely related to the distance between the first characteristic point and the key point of the grid region;
the calculating a local homography matrix of each grid region in the image to be registered according to the first feature point set, the second feature point set and the matching relation comprises the following steps:
for each grid region in the image to be registered, calculating a local homography matrix of the grid region according to the weight matrix of the grid region, the first characteristic point set, the second characteristic point set and the matching relation;
Determining a foreground target area in each grid area;
for any grid area, taking the central points of all pixels in the foreground target area of the grid area as key points of the grid area under the condition that the grid area comprises the foreground target area; for any grid area, taking the central points of all pixels in the grid area as key points of the grid area under the condition that the grid area does not comprise a foreground target area;
for each grid region in the image to be registered, calculating a local homography matrix of the grid region according to the weight matrix of the grid region, the first feature point set, the second feature point set and the matching relationship, wherein the local homography matrix comprises:
determining a feature point matrix according to the first feature point set, the second feature point set and the matching relation;
for each grid region in the image to be registered, multiplying the weight matrix of the grid region with the characteristic point matrix to obtain a corrected matrix;
and carrying out singular value decomposition on the corrected matrix to obtain a local homography matrix of the grid region.
2. The method according to claim 1, wherein the stitching the mapping areas of each grid area to obtain the target image after the registration of the image to be registered includes:
And splicing the mapping areas of the grid areas, and smoothing pixels at the spliced positions to obtain the target image after registration of the images to be registered.
3. The method according to claim 1 or 2, wherein the calculating weights of the first feature points with respect to the grid area for each grid area of the image to be registered, respectively, to obtain a weight matrix of the grid area, includes:
for each grid region of the image to be registered, determining the weight of each first feature point relative to the grid region based on the distance between each first feature point and the key point of the grid region;
and determining a weight matrix of each grid region of the image to be registered according to the weight of each first characteristic point relative to the grid region.
4. An image registration apparatus, the apparatus comprising:
the image acquisition module is used for acquiring an image to be registered and a reference image, wherein the image to be registered and the reference image comprise the same target;
the feature point acquisition module is used for carrying out feature point detection and feature point matching on the image to be registered and the reference image to obtain a first feature point set of the image to be registered, a second feature point set of the reference image, and a matching relation between each first feature point in the first feature point set and each second feature point in the second feature point set, wherein the first feature point and the second feature point with the matching relation represent the same point in the same target;
The grid region dividing module is used for dividing the image to be registered into a plurality of grid regions;
the local homography matrix determining module is used for calculating a local homography matrix of each grid area in the image to be registered according to the first characteristic point set, the second characteristic point set and the matching relation;
the grid region mapping module is used for mapping pixels in each grid region in the image to be registered by utilizing a local homography matrix of the grid region to obtain a mapping region of the grid region;
the mapping region splicing module is used for splicing the mapping regions of the grid regions to obtain the target image after the registration of the images to be registered;
the weight matrix determining module is used for respectively calculating the weight of each first characteristic point relative to each grid area of the image to be registered so as to obtain a weight matrix of the grid area, wherein the weight matrix of the grid area consists of the weight of each first characteristic point relative to the grid area, and the weight of the first characteristic point relative to the grid area and the distance between the first characteristic point and the key point of the grid area are inversely related to any first characteristic point;
The local homography matrix determining module is specifically configured to: for each grid region in the image to be registered, calculating a local homography matrix of the grid region according to the weight matrix of the grid region, the first characteristic point set, the second characteristic point set and the matching relation;
the key point determining module is used for determining foreground target areas in all grid areas; for any grid area, taking the central points of all pixels in the foreground target area of the grid area as key points of the grid area under the condition that the grid area comprises the foreground target area; for any grid area, taking the central points of all pixels in the grid area as key points of the grid area under the condition that the grid area does not comprise a foreground target area;
the local homography matrix determining module is specifically configured to: determining a feature point matrix according to the first feature point set, the second feature point set and the matching relation; for each grid region in the image to be registered, multiplying the weight matrix of the grid region with the characteristic point matrix to obtain a corrected matrix; and carrying out singular value decomposition on the corrected matrix to obtain a local homography matrix of the grid region.
5. The apparatus of claim 4, wherein the map region stitching module is specifically configured to:
and splicing the mapping areas of the grid areas, and smoothing pixels at the spliced positions to obtain the target image after registration of the images to be registered.
6. The apparatus according to claim 4 or 5, wherein the weight matrix determining module is specifically configured to: for each grid region of the image to be registered, determining the weight of each first feature point relative to the grid region based on the distance between each first feature point and the key point of the grid region; and determining a weight matrix of each grid region of the image to be registered according to the weight of each first characteristic point relative to the grid region.
7. An electronic device, comprising a processor and a memory;
the memory is used for storing a computer program;
the processor is configured to implement the image registration method of any one of claims 1 to 3 when executing the program stored on the memory.
8. A computer readable storage medium, characterized in that the computer readable storage medium has stored therein a computer program which, when executed by a processor, implements the image registration method of any of claims 1-3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110260786.0A CN112927276B (en) | 2021-03-10 | 2021-03-10 | Image registration method, device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110260786.0A CN112927276B (en) | 2021-03-10 | 2021-03-10 | Image registration method, device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112927276A CN112927276A (en) | 2021-06-08 |
| CN112927276B true CN112927276B (en) | 2024-03-12 |
Family
ID=76172388
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110260786.0A Active CN112927276B (en) | 2021-03-10 | 2021-03-10 | Image registration method, device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112927276B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114666553B (en) * | 2022-05-18 | 2022-09-23 | 深圳酷源数联科技有限公司 | Coal mine underground large-visual-angle security monitoring system |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2012049098A1 (en) * | 2010-10-15 | 2012-04-19 | Iee International Electronics & Engineering S.A. | Range image pixel matching method |
| CN108280805A (en) * | 2018-01-30 | 2018-07-13 | 北京理工大学 | A kind of image split-joint method based on manifold optimization |
| CN108537830A (en) * | 2017-03-02 | 2018-09-14 | 广州康昕瑞基因健康科技有限公司 | Method for registering images and system and image taking alignment method and system |
| CN109741240A (en) * | 2018-12-25 | 2019-05-10 | 常熟理工学院 | A kind of more flat image joining methods based on hierarchical clustering |
| CN110458870A (en) * | 2019-07-05 | 2019-11-15 | 北京迈格威科技有限公司 | A kind of image registration, fusion, occlusion detection method, apparatus and electronic equipment |
| CN110781903A (en) * | 2019-10-12 | 2020-02-11 | 中国地质大学（武汉） | Unmanned aerial vehicle image splicing method based on grid optimization and global similarity constraint |
| CN111105350A (en) * | 2019-11-25 | 2020-05-05 | 南京大学 | Real-time video splicing method based on self homography transformation under large parallax scene |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10229347B2 (en) * | 2017-05-14 | 2019-03-12 | International Business Machines Corporation | Systems and methods for identifying a target object in an image |
-
2021
- 2021-03-10 CN CN202110260786.0A patent/CN112927276B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2012049098A1 (en) * | 2010-10-15 | 2012-04-19 | Iee International Electronics & Engineering S.A. | Range image pixel matching method |
| CN108537830A (en) * | 2017-03-02 | 2018-09-14 | 广州康昕瑞基因健康科技有限公司 | Method for registering images and system and image taking alignment method and system |
| CN108280805A (en) * | 2018-01-30 | 2018-07-13 | 北京理工大学 | A kind of image split-joint method based on manifold optimization |
| CN109741240A (en) * | 2018-12-25 | 2019-05-10 | 常熟理工学院 | A kind of more flat image joining methods based on hierarchical clustering |
| CN110458870A (en) * | 2019-07-05 | 2019-11-15 | 北京迈格威科技有限公司 | A kind of image registration, fusion, occlusion detection method, apparatus and electronic equipment |
| CN110781903A (en) * | 2019-10-12 | 2020-02-11 | 中国地质大学（武汉） | Unmanned aerial vehicle image splicing method based on grid optimization and global similarity constraint |
| CN111105350A (en) * | 2019-11-25 | 2020-05-05 | 南京大学 | Real-time video splicing method based on self homography transformation under large parallax scene |
Non-Patent Citations (3)
| Title |
|---|
| Adaptive As-Natural-As-Possible Image Stitching;Chung-Ching Lin 等;《2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)》;第1156-1160页 * |
| Asprojective-as-possible image stitching with moving DLT;J. Zaragoza等;《In Computer Vision and Pattern Recognition (CVPR)》;第1289-1292段 * |
| 基于双目视觉的自由曲面三维重建研究;方程骏;《国优秀硕士学位论文全文数据库 信息科技辑》;20160115;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112927276A (en) | 2021-06-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109035320B (en) | Monocular vision-based depth extraction method | |
| CN109146980B (en) | Monocular vision based optimized depth extraction and passive distance measurement method | |
| CN110223226B (en) | Panoramic image splicing method and system | |
| CN110969670B (en) | Multispectral camera dynamic three-dimensional calibration method based on significant features | |
| CN109784250B (en) | Positioning method and device of automatic guide trolley | |
| CN112270719B (en) | Camera calibration method, device and system | |
| CN111862224A (en) | Method and device for determining external parameters between camera and laser radar | |
| CN109828250B (en) | Radar calibration method, calibration device and terminal equipment | |
| CN113592957A (en) | Multi-laser radar and multi-camera combined calibration method and system | |
| CN113421305A (en) | Target detection method, device, system, electronic equipment and storage medium | |
| CN109255818B (en) | Novel target and extraction method of sub-pixel level angular points thereof | |
| CN112927306B (en) | Calibration method and device of shooting device and terminal equipment | |
| CN112308930B (en) | Camera external parameter calibration method, system and device | |
| CN112465877A (en) | Kalman filtering visual tracking stabilization method based on motion state estimation | |
| CN111583119B (en) | Orthoimage splicing method and equipment and computer readable medium | |
| CN111383254A (en) | Depth information acquisition method and system and terminal equipment | |
| CN109598674B (en) | Image stitching method and device | |
| CN112927276B (en) | Image registration method, device, electronic equipment and storage medium | |
| CN108520541B (en) | Calibration method of wide-angle camera | |
| CN111383264A (en) | Positioning method, positioning device, terminal and computer storage medium | |
| CN113379845A (en) | Camera calibration method and device, electronic equipment and storage medium | |
| CN116679267A (en) | Combined calibration method, device, equipment and storage medium based on radar and image | |
| CN116977671A (en) | Target tracking method, device, equipment and storage medium based on image space positioning | |
| CN112102378A (en) | Image registration method and device, terminal equipment and computer readable storage medium | |
| CN116630216A (en) | Target fusion method, device, equipment and storage medium based on radar and image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |