CN117036758A - Two-dimensional image target matching method, electronic device and storage medium - Google Patents
Two-dimensional image target matching method, electronic device and storage medium Download PDFInfo
- Publication number
- CN117036758A CN117036758A CN202311303760.5A CN202311303760A CN117036758A CN 117036758 A CN117036758 A CN 117036758A CN 202311303760 A CN202311303760 A CN 202311303760A CN 117036758 A CN117036758 A CN 117036758A
- Authority
- CN
- China
- Prior art keywords
- image
- dimensional
- point
- dimensional feature
- point cloud
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 61
- 238000003860 storage Methods 0.000 title claims abstract description 13
- 230000009466 transformation Effects 0.000 claims abstract description 78
- 239000011159 matrix material Substances 0.000 claims abstract description 60
- 238000004590 computer program Methods 0.000 claims description 14
- 238000013519 translation Methods 0.000 claims description 8
- 238000000844 transformation Methods 0.000 claims description 5
- 230000001131 transforming effect Effects 0.000 claims description 4
- 238000010276 construction Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 abstract description 2
- 238000004422 calculation algorithm Methods 0.000 description 18
- 238000000605 extraction Methods 0.000 description 11
- 238000004891 communication Methods 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 5
- 238000013135 deep learning Methods 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 230000001815 facial effect Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000012937 correction Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 210000005069 ears Anatomy 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000036039 immunity Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000005303 weighing Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
Abstract
The application discloses a two-dimensional image target matching method, electronic equipment and a storage medium, which belong to the technical field of image processing, and the two-dimensional image target matching method provided by the embodiment of the application comprises the following steps: extracting a plurality of first image two-dimensional feature points from the first image; constructing a first image three-dimensional feature point corresponding to each first image two-dimensional feature point based on each first image two-dimensional feature point and first feature information corresponding to each first image two-dimensional feature point; constructing a first point cloud based on all the first image three-dimensional feature points; constructing a second point cloud corresponding to the second image according to the same method; determining a transformation matrix between the first point cloud and the second point cloud; based on the transformation matrix, a matching location of the target region of the first image in the second image is determined.
Description
Technical Field
The present application relates to the field of image processing technologies, and in particular, to a two-dimensional image target matching method, an electronic device, and a storage medium.
Background
Two-Dimensional (2D) image object matching generally solves the following problems: knowing the position of the target in image a, the position of the target in the image B to be inspected is found.
To achieve this goal, it is necessary to uniquely (in practice as close as possible, and difficult to truly uniquely) characterize the target with a set of information, which involves feature extraction. Compared with global feature extraction, the computing amount of local feature extraction is small, but because the relative position relation among feature set elements is ignored, mismatching pairs are easy to occur when 2D image matching is performed based on local feature points at present.
Disclosure of Invention
The application provides a two-dimensional image target matching method, electronic equipment and a storage medium, which are used for solving the problem of low target matching quality when 2D image matching is performed based on local feature points.
In a first aspect, the present application provides a two-dimensional image object matching method, including:
extracting a plurality of first image two-dimensional feature points from the first image;
constructing a first image three-dimensional feature point corresponding to each first image two-dimensional feature point based on each first image two-dimensional feature point and first feature information corresponding to each first image two-dimensional feature point;
constructing a first point cloud based on all the first image three-dimensional feature points;
extracting a plurality of second image two-dimensional feature points from the second image;
Constructing a second image three-dimensional feature point corresponding to each second image two-dimensional feature point based on each second image two-dimensional feature point and first feature information corresponding to each second image two-dimensional feature point;
constructing a second point cloud based on all the second image three-dimensional feature points;
determining a transformation matrix between the first point cloud and the second point cloud;
based on the transformation matrix, a matching location of a target region of the first image in the second image is determined.
In some embodiments, the extracting the plurality of first image two-dimensional feature points at the first image includes:
performing target identification in the first image, and determining a target area of the first image;
and extracting a plurality of two-dimensional characteristic points of the first image in a target area of the first image.
In some embodiments, the extracting the plurality of second image two-dimensional feature points at the second image includes:
predicting the motion trail of the target in the first image and determining the target area of the second image;
and extracting a plurality of second image two-dimensional feature points in a target area of the second image.
In some embodiments, the method for constructing any three-dimensional feature point of the first image three-dimensional feature point or the second image three-dimensional feature point includes:
Obtaining a first characteristic value of the two-dimensional characteristic point based on first characteristic information of the two-dimensional characteristic point corresponding to the three-dimensional characteristic point and the size of an image where the two-dimensional characteristic point is located;
and constructing the three-dimensional feature points by taking the image coordinates of the two-dimensional feature points and the first feature values of the two-dimensional feature points as three-dimensional coordinates.
In some embodiments, the obtaining the first feature value of the two-dimensional feature point based on the first feature information of the two-dimensional feature point corresponding to the three-dimensional feature point and the size of the image where the two-dimensional feature point is located includes:
obtaining a first characteristic value of the two-dimensional characteristic point based on the following formula:
wherein,for a first eigenvalue of the two-dimensional eigenvalue, and (2)>For the data value of the first characteristic information of the two-dimensional characteristic point, < >>For the image width of the image where the two-dimensional feature point is located,/->For the image height of the image where the two-dimensional feature points are located，/>And the maximum value in the value range of the first characteristic information of the two-dimensional characteristic points is obtained.
In some embodiments, the determining a transformation matrix between the first point cloud and the second point cloud comprises:
determining a transformation matrix between the first point cloud and the second point cloud based on the following formula:
Wherein,for a transformation matrix between the first point cloud and the second point cloud +.>Representing a rotation transformation in said transformation matrix, < >>Representing a translation transformation in said transformation matrix, < >>Representing rotation transformation +.>Representing translation transformations +.>Representing three-dimensional coordinates of an ith three-dimensional feature point in the second point cloud, +.>Representing three-dimensional coordinates of an ith three-dimensional feature point in the first point cloud, +.>For the number of three-dimensional feature points in the first point cloud, < > the first point cloud>And the number of the three-dimensional characteristic points in the second point cloud is the number of the three-dimensional characteristic points.
In some embodiments, the determining, based on the transformation matrix, a matching location of a target region of the first image in the second image comprises:
transforming the first point cloud based on the transformation matrix to obtain a third point cloud;
and determining the matching position of the target area of the first image in the second image based on the two-dimensional image coordinates of each three-dimensional characteristic point in the third point cloud.
In some embodiments, the first characteristic information includes:
image gray information; or,
hue H information in hue saturation value HSV.
In some embodiments, the predicting the motion trajectory of the object in the first image includes:
And predicting the motion trail of the target in the first image based on the maximum motion speed of the target in the first image and the shooting time interval between the first image and the second image.
In some embodiments, the method further comprises:
predicting the motion trail of a target in the P-1 frame image, and determining the target area of the P frame image;
extracting a plurality of two-dimensional characteristic points of the P-th frame image in a target area of the P-th frame image;
constructing a P-frame image three-dimensional feature point corresponding to each P-frame image two-dimensional feature point based on each P-frame image two-dimensional feature point and first feature information corresponding to each P-frame image two-dimensional feature point;
constructing a P point cloud based on all the three-dimensional characteristic points of the P frame image;
determining a transformation matrix between the first point cloud and the P-th point cloud;
determining a matching position of a target area of the first image in the P frame image based on the transformation matrix;
wherein the first image is a first frame image, and P is an integer greater than or equal to 2.
In a second aspect, the application provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing a method as described in any one of the preceding claims when executing the program.
In a third aspect, the application provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a method as described in any of the above.
In a fourth aspect, the application provides a computer program product comprising a computer program which, when executed by a processor, implements a method as described in any of the above.
According to the two-dimensional image target matching method, the three-dimensional feature points of the first image and the second image and the first point cloud and the second point cloud are constructed, so that the transformation matrix between the first point cloud and the second point cloud is determined, and the matching position of the target area of the first image in the second image is determined according to the transformation matrix, so that the 3D image matching algorithm is applied to 2D image target matching, and the accuracy of an image matching result is improved.
Drawings
In order to more clearly illustrate the application or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the application, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a two-dimensional image object matching method according to an embodiment of the present application;
FIG. 2 is a second flow chart of a two-dimensional image object matching method according to an embodiment of the application;
fig. 3 is a schematic physical structure of an electronic device according to an embodiment of the present application.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the technical solutions of the present application will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
The terms first, second and the like in the description and in the claims, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the application are capable of operation in sequences other than those illustrated or otherwise described herein, and that the "first" and "second" distinguishing between objects generally are not limited in number to the extent that the first object may, for example, be one or more. Furthermore, in the description and claims, "and/or" means at least one of the connected objects, and the character "/" generally means a relationship in which the associated object is an "or" before and after.
To achieve image object matching, it is necessary to uniquely (in practice, as closely as possible, and difficult to uniquely) characterize the object with a set of information, which involves feature extraction. Features are further divided into local features and global features from a range of features.
1. Local features
The object is represented by a set of components using local information representative of the image. For example, for a face, eyes, ears, nose, and mouth are local features of the face, and for convenience of explanation, the face feature set= { left eye, right eye, left ear, right ear, nose, and mouth }. If in image a the set of facial features of one person has been calculated, in the other image B it is only necessary to determine if there are also features in the set of facial features, and it is possible to determine if the face of this person is present in image B, and where it is present.
For computer programs, the selection of local features is required in a suitable manner to ensure that the local features that are taken are representative and reliable, and Scale-invariant feature transform (SIFT) operators are very representative algorithms for achieving this function.
The using method of the local features in target matching comprises the following steps: the local feature sets are calculated and then matched one by one, so that the position of a new target can be determined.
The advantages are that: the local feature extraction is small in calculation amount and is representative local information.
Disadvantages: local feature matching ignores the relative positional relationship between the feature set elements, and local information may have similarity, so that a mismatch pair is likely to occur when a local feature set is matched. In order to cope with this problem, an optimization method is often adopted, so that the influence of a small number of mismatching pairs can be eliminated, but a large number of mismatching pairs are not solved.
2. Global features
The object is extracted as a whole. The commonly used feature templates belong to this class of methods. The most commonly used deep learning is to perform operations such as multi-layer convolution and the like to fuse local information layer by layer to obtain global information.
The advantages are that: the matching precision is high, and the mismatching rate is low.
Disadvantages: the calculation amount is particularly large.
The local image feature matching ignores the relative position relation between the features, and the current global feature matching feature dimension is large, and the calculated amount is obviously higher than that of the local feature matching.
In three-dimensional (Three Dimensional, 3D) image object matching, an iterative closest point (Iterative Closest Point, ICP) algorithm is typically used to register 2 point clouds, i.e. point cloud 1 is rotated R and translated t such that point cloud 1 and point cloud 2 coincide as much as possible. If a part of the point cloud 2 (this part is denoted as B) is as apparent as the point cloud 1 and the position is not determined, the point cloud 1 will coincide with B after ICP registration, and this transformation matrix R, t can be obtained. That is, knowing the point cloud 1, the point cloud 2, and [ R, t ] obtained by ICP registration, B can be found, that is, the position of the target of the point cloud 1 in the point cloud 2 is found. As can be seen, the ICP algorithm is for a 3D set of spatial points and is not applicable to a 2D set of spatial points.
Therefore, the application provides a 2D image target matching method, wherein a 3D registration algorithm is used for 2D image target matching, namely matching is performed in a mode of 'local feature points+ICP'. The ICP cannot directly process the 2D image, where three-dimensional feature points (x, y, f) are constructed by adding the position coordinates (x, y) of the local feature points and the one-dimensional normalized feature value f, so that a plurality of feature points obtain a 3D point cloud, and the 2D image registration problem can be solved by using a 3D point cloud registration method. Then, the ICP algorithm is used for calculating a transformation matrix [ R, t ], and the transformation matrix is converted back into the 2D image space to obtain the position of the target in the new image. The used feature point set is a local feature, so that the calculated amount is not large, but the local feature point set is matched as a whole, and the global matching effect is realized; and the feature points are sparse, each point has only 3 dimensions (xyz), so the calculation amount of the matching process is not large, and the matching precision of 2D image target matching can be improved under the condition of not large calculation amount.
Fig. 1 is a schematic flow chart of a two-dimensional image target matching method according to an embodiment of the present application. As shown in fig. 1, there is provided a two-dimensional image object matching method including the steps of: step 110, step 120, step 130, step 140, step 150, step 160, step 170, step 180. The method flow steps are only one possible implementation of the application.
Step 110, extracting a plurality of first image two-dimensional feature points from the first image.
Specifically, the first image may be a first frame image in a video, or may be any image containing an identifiable object. The plurality of first image two-dimensional feature points may be extracted from the whole image, or may be extracted from a partial region of the first image. The feature extraction method may be appropriately selected according to the characteristics of the first image, for example, a feature extraction method such as a speeded-up-up robust features (SURF) algorithm, a direction gradient histogram (Histogram of Oriented Gradient, HOG) algorithm, or a SIFT algorithm. The plurality of first image two-dimensional feature points extracted in the first image may be represented by two-dimensional coordinates of the feature points in the first image. For example, a plurality of two-dimensional feature points may be extracted in a first frame image of a piece of video, and these feature points may be represented by two-dimensional coordinates of these feature points in the first frame image.
In some embodiments, extracting the plurality of first image two-dimensional feature points at the first image may include:
performing target identification in the first image, and determining a target area of the first image;
a plurality of first image two-dimensional feature points are extracted in a target area of the first image.
In one embodiment, the object recognition is performed in the first image, which may be image recognition performed in the first image, for example, by a deep learning method, or by a conventional image recognition method. By performing object recognition in the first image, a target area of the first image can be obtained. A plurality of first image two-dimensional feature points are then extracted in a target region of the first image. For example, image recognition may be performed in a first frame image of a motion video of a certain object, an area where the object is located in the first frame image is determined as a target area, and then a plurality of two-dimensional feature points are extracted in the target area.
Step 120, based on each of the first image two-dimensional feature points and the first feature information corresponding to each of the first image two-dimensional feature points, a first image three-dimensional feature point corresponding to each of the first image two-dimensional feature points is constructed.
Specifically, after extracting the plurality of first image two-dimensional feature points, first feature information corresponding to each first image two-dimensional feature point may also be determined.
In some embodiments, the first characteristic information may include:
image gray information; or,
hue H information in hue saturation values (Hue Saturation Value, HSV).
In one embodiment, when the first feature information is image gray information, the first feature information corresponding to the two-dimensional feature point of the first image may be determined according to a color gamut (RGB) component value of each two-dimensional feature point of the first image, for example, calculated by a Gamma correction algorithm.
In one embodiment, when the first feature information is hue H information in HSV, the first feature information corresponding to each of the two-dimensional feature points of the first image may be determined according to the RGB component value of the first two-dimensional feature point, for example, by an existing algorithm for converting the RGB component into the HSV component.
The first feature information may also be other image feature information, such as texture features, shape features, spatial relationship features, and the like.
After determining the first feature information corresponding to each first image two-dimensional feature point, the first feature information can be normalized to an order of magnitude similar to the size of the first image, and then the first image three-dimensional feature points corresponding to the first image two-dimensional feature points are constructed by combining the first image two-dimensional feature points.
And 130, constructing a first point cloud based on all the first image three-dimensional feature points.
Specifically, after the construction of all the first image three-dimensional feature points is completed, the first point cloud may be constructed according to the three-dimensional coordinate values of each first image three-dimensional feature point.
And 140, extracting a plurality of second image two-dimensional feature points from the second image.
Specifically, the second image may be an M-th frame image (M is greater than or equal to 2) in the video in which the first image is located, or may be an image in which an identifiable object in the first image is photographed after the photographing time of the first image. The plurality of second image two-dimensional feature points may be extracted in the second image by the same method as that used in the first image. The plurality of second image two-dimensional feature points may be extracted from the entire image, or may be extracted from a part of the second image. For example, a plurality of two-dimensional feature points may be extracted in a second frame image of a video.
In some embodiments, extracting the plurality of second image two-dimensional feature points at the second image may include:
Predicting a motion trail of a target in the first image, and determining a target area of the second image;
a plurality of second image two-dimensional feature points are extracted in a target area of the second image.
In one embodiment, the motion trail of the target in the first image may be predicted according to some information (such as speed, direction, etc.) when the target in the first image moves, so as to determine the approximate area of the target in the first image in the second image, thereby determining the target area of the second image. A plurality of second image two-dimensional feature points are then extracted in a target region of the second image. For example, the second image is a second frame image in a video, after determining the motion speed and direction of the target in the first frame image in the video, the target area of the second frame image may be determined by predicting the area in the first frame image where the target may appear in the second frame image according to the shooting time difference between the second frame image and the first frame image and the motion speed and direction of the target in the first frame image, and then extracting a plurality of two-dimensional feature points in the target area.
And step 150, constructing a second image three-dimensional characteristic point corresponding to each second image two-dimensional characteristic point based on each second image two-dimensional characteristic point and the first characteristic information corresponding to each second image two-dimensional characteristic point.
Specifically, after extracting the plurality of second image two-dimensional feature points, the first feature information corresponding to each second image two-dimensional feature point may also be determined.
After determining the first feature information corresponding to each second image two-dimensional feature point, the first feature information can be normalized to an order of magnitude similar to the size of the second image, and then the second image two-dimensional feature points are combined to construct second image three-dimensional feature points corresponding to the second image two-dimensional feature points. In a different embodiment, the first feature information corresponding to the two-dimensional feature point of the second image is matched with the first feature information corresponding to the two-dimensional feature point of the first image. For example, if the first feature information corresponding to the two-dimensional feature point of the first image is the image gray information, the first feature information corresponding to the two-dimensional feature point of the second image is also the image gray information; if the first characteristic information corresponding to the two-dimensional characteristic points of the first image is hue H information in HSV, the first characteristic information corresponding to the two-dimensional characteristic points of the second image is also hue H information in HSV.
And 160, constructing a second point cloud based on all the second image three-dimensional characteristic points.
Specifically, after the construction of all the second image three-dimensional feature points is completed, the second point cloud may be constructed according to the three-dimensional coordinate values of each second image three-dimensional feature point.
Step 170, determining a transformation matrix between the first point cloud and the second point cloud.
In particular, a transformation matrix between a first point cloud, which corresponds to a source point cloud, and a second point cloud, which corresponds to a target point cloud, i.e., an optimal transformation of the first point cloud to the second point cloud, may be determined according to an existing point cloud registration (Point Cloud Registration) algorithm (e.g., ICP algorithm, normal distribution transformation (Normal Distributions Transform, NDT) algorithm, etc.).
Step 180, determining a matching position of the target area of the first image in the second image based on the transformation matrix.
Specifically, after a transformation matrix between the first point cloud and the second point cloud is determined, the transformation matrix may be used to transform the first point cloud, and then, according to the three-dimensional feature points obtained after the transformation of the three-dimensional feature points of the first image in the target area of the first image, a matching position of the target area of the first image in the second image is determined.
In some embodiments, determining a matching location of the target region of the first image in the second image based on the transformation matrix may include:
transforming the first point cloud based on the transformation matrix to obtain a third point cloud;
And determining the matching position of the target area of the first image in the second image based on the two-dimensional image coordinates of each three-dimensional characteristic point in the third point cloud.
For example, the matching location of the target region of the first image in the second image may be determined according to the following formula:
wherein,representing a third point cloud->Representing a transformation matrix between a first point cloud and a second point cloud,/a>Representing a first point cloud->Representing the first dimension coordinates of the ith three-dimensional feature point in the third point cloud, +.>Representing the second dimension coordinates of the ith three-dimensional feature point in the third point cloud, +.>Third-dimensional coordinates representing an ith three-dimensional feature point in the third point cloud, ++>Three-dimensional in a first point cloudThe number of feature points. In one embodiment, the first 2-dimensional coordinates (m xi， m yi ) The location in the second image to which the target area is matched.
According to the first dimensional coordinates and the second dimensional coordinates of the three-dimensional feature points obtained by transforming the three-dimensional feature points of the first image in the target area of the first image, the matching position of the target area of the first image in the second image can be determined.
In some embodiments, in the case where the target in the second image is blocked, the matching error of the matching position in the second image may be relatively large, and at this time, the target area of the second image predicted from the motion trajectory of the target in the first image may be used as the matching result.
In the embodiment of the application, the three-dimensional characteristic points of the first image and the second image are constructed, the first point cloud and the second point cloud are constructed, the transformation matrix between the first point cloud and the second point cloud is further determined, and the matching position of the target area of the first image in the second image is determined according to the transformation matrix, so that the 3D image matching algorithm is applied to 2D image target matching, and the accuracy of the image matching result is improved.
It should be noted that each embodiment of the present application may be freely combined, exchanged in order, or separately executed, and does not need to rely on or rely on a fixed execution sequence.
In some embodiments, the method for constructing any three-dimensional feature point of the first image three-dimensional feature point or the second image three-dimensional feature point includes:
obtaining a first characteristic value of the two-dimensional characteristic point based on the first characteristic information of the two-dimensional characteristic point corresponding to the three-dimensional characteristic point and the size of the image where the two-dimensional characteristic point is located;
and constructing the three-dimensional feature points by taking the image coordinates of the two-dimensional feature points and the first feature values of the two-dimensional feature points as three-dimensional coordinates.
Specifically, when constructing a three-dimensional feature point of the first image or a three-dimensional feature point of the second image, a first feature value of the two-dimensional feature point may be calculated according to first feature information of the two-dimensional feature point corresponding to the three-dimensional feature point and a size of an image in which the two-dimensional feature point is located.
In some embodiments, the first eigenvalue of the two-dimensional eigenvalue may be derived based on the following formula:
wherein,is the first characteristic value of the two-dimensional characteristic point, < >>The data value of the first feature information being a two-dimensional feature point,for the image width of the image in which the two-dimensional feature point is located, < >>For the image height of the image in which the two-dimensional feature point is located, < >>Is the maximum value in the value range of the first characteristic information of the two-dimensional characteristic point.
When the first characteristic information of the two-dimensional characteristic point is image gradation information, for example, the value range of the first characteristic information is 0 to 255,is 255.
For another example, when the first characteristic information of the two-dimensional characteristic point is chromaticity information, the value of the first characteristic information ranges from 0 to 179,is 179.
After the first feature value of the two-dimensional feature point is determined, the image coordinates of the two-dimensional feature point and the first feature value of the two-dimensional feature point can be used as the three-dimensional coordinates of the corresponding three-dimensional feature point, so that the three-dimensional feature point is constructed.
In some embodiments, determining a transformation matrix between the first point cloud and the second point cloud may include:
a transformation matrix between the first point cloud and the second point cloud is determined according to an ICP algorithm.
In one embodiment, the transformation matrix between the first point cloud and the second point cloud may be determined based on the following formula:
Wherein,for a transformation matrix between a first point cloud and a second point cloud +.>Representing the rotation transform in the transform matrix, +.>Representing a translation transformation in a transformation matrix, +.>Representing rotation transformation +.>Representing translation transformations +.>Representing three-dimensional coordinates of an ith three-dimensional feature point in the second point cloud,/for>Showing the three-dimensional coordinates of the ith three-dimensional feature point in the first point cloud, +.>For the number of three-dimensional feature points in the first point cloud, < >>Is the second pointThe number of three-dimensional feature points in the cloud.
In some embodiments, predicting a motion trajectory of an object in a first image includes:
and predicting the motion trail of the target in the first image based on the maximum motion speed of the target in the first image and the shooting time interval between the first image and the second image.
Specifically, when predicting the motion trajectory of the object in the first image and determining the object region of the second image, the motion trajectory of the object in the first image during the photographing time interval may be predicted based on the maximum motion speed of the object in the first image and the photographing time interval between the first image and the second image, so that the approximate region of the object in the first image in the second image, that is, the object region of the second image, is predicted. For example, in the case where the first image is a first frame image in a video and the second image is a second frame image in the video, a region of the first frame image in the second frame image, that is, a target region of the second frame image, can be predicted based on the maximum moving speed of the target in the first frame image and the photographing time interval between the first frame image and the second frame image.
In some embodiments, the method further comprises:
predicting the motion trail of a target in the P-1 frame image, and determining the target area of the P frame image;
extracting a plurality of two-dimensional characteristic points of the P-th frame image in a target area of the P-th frame image;
constructing a P-frame image three-dimensional feature point corresponding to each P-frame image two-dimensional feature point based on each P-frame image two-dimensional feature point and first feature information corresponding to each P-frame image two-dimensional feature point;
constructing a P-th point cloud based on all the three-dimensional characteristic points of the P-th frame image;
determining a transformation matrix between the first point cloud and the P-th point cloud;
determining a matching position of a target area of the first image in the P-frame image based on the transformation matrix;
wherein the first image is a first frame image, and P is an integer greater than or equal to 2.
Specifically, in the case of performing two-dimensional image object matching on multiple frame images in a video, image recognition (P is an integer greater than or equal to 2) may be performed on a P-1 frame image, a target area of the P-1 frame image is determined, and then the target area of the P-1 frame image is predicted according to a motion track of a target in the P-1 frame image.
In one embodiment, after the P-1 frame image is identified to determine the target region of the P-1 frame image, the step of calculating the transformation matrix is not required.
In one embodiment, the target region in the image may be re-identified at regular intervals, and then the target region in the subsequent frame image may be predicted according to the re-identified target region in the image.
For example, every 10 frames of images are re-identified, namely, the target area is determined by adopting an image identification mode for the 1 st frame of images, the 2 nd to 10 th frames of images adopt the target matching method provided by the application, the 11 th frame of images adopt an image identification mode for determining the target area, the 12 th to 20 th frames of images adopt the target matching method provided by the application, and so on.
It is emphasized that, although the target region of the subsequent frame image is not predicted from the target region of the first frame image, when the target matching is performed, the transformation matrix needs to be calculated from the first point cloud, and each frame image needs to be separately calculated.
After determining the target region of the P-th frame image, a plurality of P-th frame image two-dimensional feature points may be extracted in the target region of the P-th frame image, and then a P-th frame image three-dimensional feature point corresponding to each P-th frame image two-dimensional feature point may be constructed according to each P-th frame image two-dimensional feature point and first feature information corresponding to each P-th frame image two-dimensional feature point.
Based on all the three-dimensional feature points of the P-th frame image, a P-th point cloud can be constructed, then a transformation matrix between the first point cloud and the P-th point cloud is calculated, and the matching position of the target area of the first image in the P-th frame image is determined according to the transformation matrix.
The two-dimensional image target matching method provided by the embodiment of the application is illustrated below, so that the technical schemes described in the embodiments of the application can be more clearly understood.
Fig. 2 is a second flowchart of a two-dimensional image object matching method according to an embodiment of the present application. As shown in fig. 2, the method comprises the steps of:
1a, detecting a target area D in an image A, and then extracting a plurality of characteristic points in the target area D, wherein the images of the characteristic points are marked as. The method of feature extraction here is appropriately selected according to the target characteristics, and may be SURF, HOG, or the like.
1b, calculating one-dimensional characteristic values of the characteristic point set：
Wherein,is the serial number of the feature point, < >>Representing characteristic points->Gray value of +.>And->Representing the width and height of image a, respectively.
1c, toAs->Constructing a three-dimensional point +.>. Each characteristic point is processed according to the method to obtain a three-dimensional point cloud:
2. extracting a plurality of characteristic points from the image B by using the same characteristic extraction method in the step 1a, and constructing a three-dimensional point cloud in the same manner in the steps 1B and 1 c:
3a, calculating the point cloud by using ICP algorithmTo the point cloud->Is the best transformation of (a):
wherein,is a point cloud->And Point cloud->Transformation matrix between>Representing the rotational transformations in the transformation matrix,representing a translation transformation in a transformation matrix, +.>Representing rotation transformation +.>Representing translation transformations +.>Representing a point cloud->Three-dimensional coordinates of the ith three-dimensional feature point, < ->Representing a point cloud->Three-dimensional coordinates of the ith three-dimensional feature point, < ->Is a point cloud->The number of three-dimensional feature points in the middle>Is a point cloud->The number of three-dimensional feature points in the model.
3b, computing Point cloudTransformed point cloud->：
Point setThe location in image B to which object D is matched.
In step 2, a three-dimensional point cloud of the image B is extractedIn this case, the range of feature extraction can be narrowed by combining some prior information, so that feature extraction is not required for the whole image B. For example, if the object D is moving, a smaller range of the object D in the image B can be determined according to the maximum moving speed of the object and the time interval between the image A and the image B, and then the point cloud is constructed for the smaller range>This may reduce the amount of computation and reduce some interference information.
In step 1b, the image gray information is used for the one-dimensional feature F, but not limited to this, and other feature information may be used, and the information may be converted into one-dimensional data and normalized to a level similar to the image size. For example, it is also possible to feature H information of HSV, but such calculation amount may be larger than that of gray information.
According to the embodiment of the application, the 3D feature points are constructed by the 2D image feature points, so that the position coordinates and the image local feature information can be fused together, the information richness of the feature points participating in matching is increased, and the target matching quality is improved. By applying the ICP algorithm to 2D image target matching, the matching result is the optimal result of comprehensive weighing of all feature points, instead of the independent matching of all feature points, so that the method has strong noise immunity and the robustness of target matching is improved.
Fig. 3 illustrates a physical schematic diagram of an electronic device, as shown in fig. 3, where the electronic device may include: processor 310, communication interface (Communications Interface) 320, memory 330 and communication bus 340, wherein processor 310, communication interface 320, memory 330 accomplish communication with each other through communication bus 340. The processor 310 may invoke logic instructions in the memory 330 to perform a two-dimensional image object matching method comprising: extracting a plurality of first image two-dimensional feature points from the first image;
constructing a first image three-dimensional feature point corresponding to each first image two-dimensional feature point based on each first image two-dimensional feature point and first feature information corresponding to each first image two-dimensional feature point;
Constructing a first point cloud based on all the first image three-dimensional feature points;
extracting a plurality of second image two-dimensional feature points from the second image;
constructing a second image three-dimensional feature point corresponding to each second image two-dimensional feature point based on each second image two-dimensional feature point and the first feature information corresponding to each second image two-dimensional feature point;
constructing a second point cloud based on all the second image three-dimensional feature points;
determining a transformation matrix between the first point cloud and the second point cloud;
based on the transformation matrix, a matching location of the target region of the first image in the second image is determined.
Further, the logic instructions in the memory 330 described above may be implemented in the form of software functional units and may be stored in a computer-readable storage medium when sold or used as a stand-alone product. Based on this understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In another aspect, the present application also provides a computer program product, the computer program product comprising a computer program, the computer program being storable on a non-transitory computer readable storage medium, the computer program, when executed by a processor, being capable of executing the method provided by the above method embodiments, the method comprising: extracting a plurality of first image two-dimensional feature points from the first image;
constructing a first image three-dimensional feature point corresponding to each first image two-dimensional feature point based on each first image two-dimensional feature point and first feature information corresponding to each first image two-dimensional feature point;
constructing a first point cloud based on all the first image three-dimensional feature points;
extracting a plurality of second image two-dimensional feature points from the second image;
constructing a second image three-dimensional feature point corresponding to each second image two-dimensional feature point based on each second image two-dimensional feature point and the first feature information corresponding to each second image two-dimensional feature point;
constructing a second point cloud based on all the second image three-dimensional feature points;
determining a transformation matrix between the first point cloud and the second point cloud;
based on the transformation matrix, a matching location of the target region of the first image in the second image is determined.
In yet another aspect, the present application further provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, is implemented to perform the two-dimensional image object matching method provided by the above-mentioned method embodiments, the method comprising: extracting a plurality of first image two-dimensional feature points from the first image;
constructing a first image three-dimensional feature point corresponding to each first image two-dimensional feature point based on each first image two-dimensional feature point and first feature information corresponding to each first image two-dimensional feature point;
constructing a first point cloud based on all the first image three-dimensional feature points;
extracting a plurality of second image two-dimensional feature points from the second image;
constructing a second image three-dimensional feature point corresponding to each second image two-dimensional feature point based on each second image two-dimensional feature point and the first feature information corresponding to each second image two-dimensional feature point;
constructing a second point cloud based on all the second image three-dimensional feature points;
determining a transformation matrix between the first point cloud and the second point cloud;
based on the transformation matrix, a matching location of the target region of the first image in the second image is determined.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present application without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and are not limiting; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims (12)
1. A two-dimensional image object matching method, comprising:
extracting a plurality of first image two-dimensional feature points from the first image;
constructing a first image three-dimensional feature point corresponding to each first image two-dimensional feature point based on each first image two-dimensional feature point and first feature information corresponding to each first image two-dimensional feature point;
constructing a first point cloud based on all the first image three-dimensional feature points;
extracting a plurality of second image two-dimensional feature points from the second image;
constructing a second image three-dimensional feature point corresponding to each second image two-dimensional feature point based on each second image two-dimensional feature point and first feature information corresponding to each second image two-dimensional feature point;
constructing a second point cloud based on all the second image three-dimensional feature points;
determining a transformation matrix between the first point cloud and the second point cloud;
based on the transformation matrix, a matching location of a target region of the first image in the second image is determined.
2. The two-dimensional image object matching method according to claim 1, wherein said extracting a plurality of first image two-dimensional feature points in the first image comprises:
Performing target identification in the first image, and determining a target area of the first image;
and extracting a plurality of two-dimensional characteristic points of the first image in a target area of the first image.
3. The two-dimensional image object matching method according to claim 2, wherein said extracting a plurality of second image two-dimensional feature points in the second image comprises:
predicting the motion trail of the target in the first image and determining the target area of the second image;
and extracting a plurality of second image two-dimensional feature points in a target area of the second image.
4. The two-dimensional image target matching method according to claim 1, wherein the construction manner of any one of the first image three-dimensional feature point or the second image three-dimensional feature point comprises:
obtaining a first characteristic value of the two-dimensional characteristic point based on first characteristic information of the two-dimensional characteristic point corresponding to the three-dimensional characteristic point and the size of an image where the two-dimensional characteristic point is located;
and constructing the three-dimensional feature points by taking the image coordinates of the two-dimensional feature points and the first feature values of the two-dimensional feature points as three-dimensional coordinates.
5. The method for matching a two-dimensional image object according to claim 4, wherein the obtaining the first feature value of the two-dimensional feature point based on the first feature information of the two-dimensional feature point corresponding to the three-dimensional feature point and the size of the image in which the two-dimensional feature point is located includes:
Obtaining a first characteristic value of the two-dimensional characteristic point based on the following formula:
；
wherein,for a first eigenvalue of the two-dimensional eigenvalue, and (2)>For the data value of the first characteristic information of the two-dimensional characteristic point, < >>For the image width of the image where the two-dimensional feature point is located,/->For the image height of the image in which the two-dimensional feature point is located,/->And the maximum value in the value range of the first characteristic information of the two-dimensional characteristic points is obtained.
6. The two-dimensional image object matching method according to any one of claims 1 to 5, wherein said determining a transformation matrix between the first point cloud and the second point cloud comprises:
determining a transformation matrix between the first point cloud and the second point cloud based on the following formula:
；
wherein,for a transformation matrix between the first point cloud and the second point cloud +.>Representing a rotation transformation in said transformation matrix, < >>Representing a translation transformation in said transformation matrix, < >>Representing rotation transformation +.>Representing translation transformations +.>Representing +.>Three-dimensional coordinates of three-dimensional feature points, +.>Representing +.>Three-dimensional coordinates of three-dimensional feature points, +.>For the number of three-dimensional feature points in the first point cloud, < > the first point cloud >And the number of the three-dimensional characteristic points in the second point cloud is the number of the three-dimensional characteristic points.
7. The two-dimensional image object-matching method according to any one of claims 1 to 5, wherein the determining, based on the transformation matrix, a matching position of the object region of the first image in the second image comprises:
transforming the first point cloud based on the transformation matrix to obtain a third point cloud;
and determining the matching position of the target area of the first image in the second image based on the two-dimensional image coordinates of each three-dimensional characteristic point in the third point cloud.
8. The two-dimensional image object matching method according to any one of claims 1 to 5, wherein the first characteristic information includes:
image gray information; or,
hue H information in hue saturation value HSV.
9. The method for matching a two-dimensional image object according to claim 3, wherein predicting the motion trajectory of the object in the first image comprises:
and predicting the motion trail of the target in the first image based on the maximum motion speed of the target in the first image and the shooting time interval between the first image and the second image.
10. The two-dimensional image object matching method according to claim 2, characterized in that the method further comprises:
predicting the motion trail of a target in the P-1 frame image, and determining the target area of the P frame image;
extracting a plurality of two-dimensional characteristic points of the P-th frame image in a target area of the P-th frame image;
constructing a P-frame image three-dimensional feature point corresponding to each P-frame image two-dimensional feature point based on each P-frame image two-dimensional feature point and first feature information corresponding to each P-frame image two-dimensional feature point;
constructing a P point cloud based on all the three-dimensional characteristic points of the P frame image;
determining a transformation matrix between the first point cloud and the P-th point cloud;
determining a matching position of a target area of the first image in the P frame image based on the transformation matrix;
wherein the first image is a first frame image, and P is an integer greater than or equal to 2.
11. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the two-dimensional image object matching method of any one of claims 1 to 10 when the program is executed by the processor.
12. A non-transitory computer readable storage medium having stored thereon a computer program, wherein the computer program when executed by a processor implements the two-dimensional image object matching method according to any of claims 1 to 10.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311303760.5A CN117036758B (en) | 2023-10-10 | 2023-10-10 | Two-dimensional image target matching method, electronic device and storage medium |
| CN202311795074.4A CN117726838A (en) | 2023-10-10 | 2023-10-10 | Two-dimensional image target matching method, electronic device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311303760.5A CN117036758B (en) | 2023-10-10 | 2023-10-10 | Two-dimensional image target matching method, electronic device and storage medium |
Related Child Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311795074.4A Division CN117726838A (en) | 2023-10-10 | 2023-10-10 | Two-dimensional image target matching method, electronic device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117036758A true CN117036758A (en) | 2023-11-10 |
| CN117036758B CN117036758B (en) | 2024-01-12 |
Family
ID=88624916
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311795074.4A Pending CN117726838A (en) | 2023-10-10 | 2023-10-10 | Two-dimensional image target matching method, electronic device and storage medium |
| CN202311303760.5A Active CN117036758B (en) | 2023-10-10 | 2023-10-10 | Two-dimensional image target matching method, electronic device and storage medium |
Family Applications Before (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311795074.4A Pending CN117726838A (en) | 2023-10-10 | 2023-10-10 | Two-dimensional image target matching method, electronic device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (2) | CN117726838A (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2004306593A (en) * | 2003-02-26 | 2004-11-04 | Fuji Photo Film Co Ltd | Method and apparatus for forming 3-dimensional image |
| CN102790893A (en) * | 2012-07-19 | 2012-11-21 | 彩虹集团公司 | Method for achieving 2D-3D conversion based on weighted average operator algorithm |
| CN111724481A (en) * | 2020-06-24 | 2020-09-29 | 嘉应学院 | Method, device, equipment and storage medium for three-dimensional reconstruction of two-dimensional image |
| CN112634172A (en) * | 2020-12-30 | 2021-04-09 | 深圳市创想三维科技有限公司 | 3D printing method and device for generating three-dimensional model by using two-dimensional image |
| CN113362446A (en) * | 2021-05-25 | 2021-09-07 | 上海奥视达智能科技有限公司 | Method and device for reconstructing object based on point cloud data |
| CN114119864A (en) * | 2021-11-09 | 2022-03-01 | 同济大学 | Positioning method and device based on three-dimensional reconstruction and point cloud matching |
| WO2023046211A1 (en) * | 2021-09-27 | 2023-03-30 | 天远三维（天津）科技有限公司 | Photogrammetry method, apparatus and device, and storage medium |
| CN116468917A (en) * | 2023-03-17 | 2023-07-21 | 湖北星纪魅族科技有限公司 | Image processing method, electronic device and storage medium |
-
2023
- 2023-10-10 CN CN202311795074.4A patent/CN117726838A/en active Pending
- 2023-10-10 CN CN202311303760.5A patent/CN117036758B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2004306593A (en) * | 2003-02-26 | 2004-11-04 | Fuji Photo Film Co Ltd | Method and apparatus for forming 3-dimensional image |
| CN102790893A (en) * | 2012-07-19 | 2012-11-21 | 彩虹集团公司 | Method for achieving 2D-3D conversion based on weighted average operator algorithm |
| CN111724481A (en) * | 2020-06-24 | 2020-09-29 | 嘉应学院 | Method, device, equipment and storage medium for three-dimensional reconstruction of two-dimensional image |
| CN112634172A (en) * | 2020-12-30 | 2021-04-09 | 深圳市创想三维科技有限公司 | 3D printing method and device for generating three-dimensional model by using two-dimensional image |
| CN113362446A (en) * | 2021-05-25 | 2021-09-07 | 上海奥视达智能科技有限公司 | Method and device for reconstructing object based on point cloud data |
| WO2023046211A1 (en) * | 2021-09-27 | 2023-03-30 | 天远三维（天津）科技有限公司 | Photogrammetry method, apparatus and device, and storage medium |
| CN114119864A (en) * | 2021-11-09 | 2022-03-01 | 同济大学 | Positioning method and device based on three-dimensional reconstruction and point cloud matching |
| CN116468917A (en) * | 2023-03-17 | 2023-07-21 | 湖北星纪魅族科技有限公司 | Image processing method, electronic device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117726838A (en) | 2024-03-19 |
| CN117036758B (en) | 2024-01-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8620085B2 (en) | Image processing apparatus, image processing method, and image processing program | |
| JP7446457B2 (en) | Image optimization method and device, computer storage medium, computer program, and electronic equipment | |
| CN109087261B (en) | Face correction method based on unlimited acquisition scene | |
| CN110400278B (en) | Full-automatic correction method, device and equipment for image color and geometric distortion | |
| CN111696196B (en) | Three-dimensional face model reconstruction method and device | |
| Mahapatra et al. | Training data independent image registration with gans using transfer learning and segmentation information | |
| CN111695462A (en) | Face recognition method, face recognition device, storage medium and server | |
| CN113469092B (en) | Character recognition model generation method, device, computer equipment and storage medium | |
| CN110930310B (en) | Panoramic image splicing method | |
| CN111199197B (en) | Image extraction method and processing equipment for face recognition | |
| Tang et al. | Graph-based tracking of the tongue contour in ultrasound sequences with adaptive temporal regularization | |
| Saleh et al. | Adaptive uncertainty distribution in deep learning for unsupervised underwater image enhancement | |
| KR102550864B1 (en) | Method and System for Object Identification by Patterns of Object Image | |
| KR101921608B1 (en) | Apparatus and method for generating depth information | |
| US11501404B2 (en) | Method and system for data processing | |
| CN109785367B (en) | Method and device for filtering foreign points in three-dimensional model tracking | |
| CN117036758B (en) | Two-dimensional image target matching method, electronic device and storage medium | |
| CN113012030A (en) | Image splicing method, device and equipment | |
| US20220198707A1 (en) | Method and apparatus with object pose estimation | |
| Ernst et al. | Check my chart: A robust color chart tracker for colorimetric camera calibration | |
| CN110751163A (en) | Target positioning method and device, computer readable storage medium and electronic equipment | |
| Vazquez-Corral et al. | Angular-based preprocessing for image denoising | |
| CN114581340A (en) | Image correction method and device | |
| CN115210758A (en) | Motion blur robust image feature matching | |
| CN113627324A (en) | Matching method and device of face images, storage medium and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |