CN112950709B - Pose prediction method, pose prediction device and robot - Google Patents
Pose prediction method, pose prediction device and robot Download PDFInfo
- Publication number
- CN112950709B CN112950709B CN202110194534.2A CN202110194534A CN112950709B CN 112950709 B CN112950709 B CN 112950709B CN 202110194534 A CN202110194534 A CN 202110194534A CN 112950709 B CN112950709 B CN 112950709B
- Authority
- CN
- China
- Prior art keywords
- image
- line
- feature
- pair
- robot
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 238000005259 measurement Methods 0.000 claims abstract description 16
- 238000000605 extraction Methods 0.000 claims description 55
- 230000006870 function Effects 0.000 claims description 19
- 238000004590 computer program Methods 0.000 claims description 18
- 230000003287 optical effect Effects 0.000 claims description 12
- 238000001514 detection method Methods 0.000 claims description 5
- 238000005457 optimization Methods 0.000 claims description 4
- 238000012545 processing Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000007781 pre-processing Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 230000008878 coupling Effects 0.000 description 4
- 238000010168 coupling process Methods 0.000 description 4
- 238000005859 coupling reaction Methods 0.000 description 4
- 239000000284 extract Substances 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000010606 normalization Methods 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 239000013589 supplement Substances 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 235000019800 disodium phosphate Nutrition 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000000047 product Substances 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/10—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration
- G01C21/12—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning
- G01C21/16—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
Landscapes
- Engineering & Computer Science (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Theoretical Computer Science (AREA)
- Automation & Control Theory (AREA)
- Image Analysis (AREA)
- Manipulator (AREA)
Abstract
The application discloses a pose prediction method, a pose prediction device, a robot and a computer readable storage medium. The method is applied to a robot provided with a binocular camera, wherein the binocular camera comprises a first camera and a second camera, and the method comprises the following steps: searching at least one pair of matched characteristic points in a first image acquired at the current moment of a first camera and a second image acquired at the current moment of a second camera; finding out at least one pair of matched line characteristics in the first image and the second image; based on at least one pair of line characteristics, at least one pair of characteristic points and inertial data output by an inertial measurement unit of the robot, a predicted pose of the robot is obtained. By the scheme of the application, more accurate and robust pose prediction of the robot can be realized.
Description
Technical Field
The application belongs to the technical field of visual algorithms, and particularly relates to a pose prediction method, a pose prediction device, a robot and a computer readable storage medium.
Background
In the process of positioning and mapping the robot, accurate pose prediction of the robot needs to be obtained. In the prior art, a constraint condition is often built by relying on a visual sensor by adopting a characteristic point method based on characteristic point matching or a direct method based on luminosity consistency, so that the pose prediction of the robot is realized. However, in some indoor scenes, such as a low-texture environment or a dark-light environment, effective feature points are difficult to extract by using a feature point method based on feature point matching, and the usability of a direct method based on consistent luminosity is also greatly reduced. That is, in weak texture and dim light scenes, the pose of the robot may not be accurately predicted.
Disclosure of Invention
The application provides a pose prediction method, a pose prediction device, a robot and a computer readable storage medium, which can realize more accurate and robust pose prediction of the robot.
In a first aspect, the present application provides a pose prediction method, where the pose prediction method is applied to a robot provided with a binocular camera, the binocular camera includes a first camera and a second camera, and the pose prediction method includes:
at least one pair of matched characteristic points are found out from a first image and a second image, wherein each pair of characteristic points comprises a first characteristic point and a second characteristic point, the first image is obtained based on an image acquired at the current moment of the first camera, the second image is obtained based on an image acquired at the current moment of the second camera, the first characteristic point is in the first image, and the second characteristic point is in the second image;
finding out at least one pair of matched line features in the first image and the second image, wherein each pair of line features comprises a first line feature and a second line feature, the first line feature is in the first image, and the second line feature is in the second image;
And obtaining a predicted pose of the robot based on the at least one pair of line features, the at least one pair of feature points, and inertial data output by an inertial measurement unit of the robot.
In a second aspect, the present application provides a pose prediction apparatus, the pose prediction apparatus being applied to a robot provided with a binocular camera, the binocular camera including a first camera and a second camera, the pose prediction apparatus comprising:
the first searching unit is used for searching at least one pair of matched characteristic points in a first image and a second image, wherein each pair of characteristic points comprises a first characteristic point and a second characteristic point, the first image is obtained based on an image acquired at the current moment of the first camera, the second image is obtained based on an image acquired at the current moment of the second camera, the first characteristic point is in the first image, and the second characteristic point is in the second image;
a second searching unit, configured to search at least one pair of matched line features in the first image and the second image, where each pair of line features includes a first line feature and a second line feature, the first line feature is in the first image, and the second line feature is in the second image;
And a prediction unit configured to obtain a predicted pose of the robot based on the at least one pair of line features, the at least one pair of feature points, and inertial data output from the inertial measurement unit of the robot.
In a third aspect, the present application provides a robot comprising a memory, a processor, a binocular camera and a computer program stored in the memory and executable on the processor, the processor implementing the steps of the method of the first aspect when executing the computer program.
In a fourth aspect, the present application provides a computer readable storage medium storing a computer program which, when executed by a processor, performs the steps of the method of the first aspect.
In a fifth aspect, the present application provides a computer program product comprising a computer program which, when executed by one or more processors, implements the steps of the method of the first aspect described above.
Compared with the prior art, the application has the beneficial effects that: for a robot provided with a first camera and a second camera (i.e. binocular cameras), at least one pair of matched characteristic points are found in a first image and a second image, wherein each pair of characteristic points comprises a first characteristic point and a second characteristic point, the first image is obtained based on an image acquired at the current moment of the first camera, the second image is obtained based on an image acquired at the current moment of the second camera, the first characteristic point is in the first image, the second characteristic point is in the second image, and at least one pair of matched line characteristics are found in the first image and the second image, each pair of line characteristics comprises a first line characteristic and a second line characteristic, the first line characteristic is in the first image, the second line characteristic is in the second image, and finally the predicted position and orientation of the robot can be obtained based on the at least one pair of characteristics, the at least one pair of characteristic points and the inertial measurement unit of the robot. The robot pose prediction method combines the characteristic points and the line characteristics to predict the pose of the robot, and can obtain the geometric structure information in the environment of the robot through the combination of the characteristic points and the line characteristics, so that the robot can realize precise and robust pose prediction under challenging weak texture and low visible light scenes. It will be appreciated that the advantages of the second to fifth aspects may be found in the relevant description of the first aspect, and are not described here again.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments or the description of the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic implementation flow chart of a pose prediction method provided by an embodiment of the present application;
fig. 2 is an exemplary diagram of two line segments before stitching provided in an embodiment of the present application;
FIG. 3 is an exemplary diagram of a segment obtained after stitching according to an embodiment of the present application;
FIG. 4 is a schematic diagram of a three-dimensional line feature re-projection residual provided by an embodiment of the present application;
fig. 5 is a schematic structural diagram of a pose prediction device according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of a robot according to an embodiment of the present application.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth such as the particular system architecture, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
In order to illustrate the technical scheme provided by the application, the following description is made by specific embodiments.
The following describes a pose prediction method provided by the embodiment of the application. The pose prediction method is applied to a robot provided with binocular cameras, and for convenience of description, one camera in the binocular cameras is a first camera, and the other camera is a second camera. For example only, the first camera may be a left-eye camera and the second camera may be a right-eye camera. Referring to fig. 1, the pose prediction method includes:
and step 101, finding out at least one pair of matched characteristic points in the first image and the second image.
In the embodiment of the application, the original images acquired by the first camera and the second camera are firstly converted into gray images by the robot, and a series of preprocessing is carried out on the gray images so as to improve the subsequent data processing efficiency. For example, the preprocessing operation may be a gaussian blur operation to reduce noise of the gray image, make the gray image smoother, and reduce the level of detail of the gray image. The preprocessed gray image can be put into use in each step of the embodiment of the application. That is, the first image refers to a gray image obtained after the preprocessing operation is performed based on the original image acquired by the first camera at the current time, and the second image refers to a gray image obtained after the preprocessing operation is performed based on the original image acquired by the second camera at the current time.
For a pair of feature points, i.e. a feature point pair, the feature point pair is composed of two feature points, a first feature point and a second feature point, wherein the first feature point is in the first image and the second feature point is in the second image. That is, one feature point in the first image and a feature point in the second image that matches the feature point constitute a pair of feature points.
In some embodiments, the robot may extract at least one feature point from the first image, and then find a feature point matching each feature point of the first image in the second image by means of optical flow matching, so as to obtain at least one pair of feature points. It should be noted that at most one feature point in the second image can only be matched with one feature point in the first image, and likewise, at most one feature point in the first image can only be matched with one feature point in the second image; that is, no duplication of feature points occurs in different pairs of feature points. In order to realize rapid extraction of the feature points, the robot first judges whether the first image obtained at present is an initial first image, namely, whether the first image is obtained based on a first frame image acquired after the first camera is started; if the first image is an initial first image, extracting feature points of the first image based on a preset first feature point extraction mode, wherein the first feature point extraction mode is related to corner points; if the first image is not the initial first image, the feature points of the first image may be extracted based on a predetermined second feature point extraction method, where the second feature point extraction method is related to optical flow. The first feature point extraction method and the second feature point extraction method are briefly described as follows:
The first feature point extraction mode refers to: and FAST (Features from Accelerated Segment Test) corner detection is carried out on the first image, and the extracted corner is taken as a characteristic point. Considering that the number of extracted corner points is usually large, and may even reach thousands, a threshold value N for the number of feature points is set here, that is, N corner points may be selected as feature points from the extracted corner points based on the response values of the respective corner points. Considering that too few feature points may cause inaccuracy of the predicted pose, too many feature points may increase the system operation amount of the robot, and thus N may be set to any integer in the interval of [70,100 ].
The second feature point extraction mode refers to: and carrying out optical flow tracking on the feature points obtained from the previous frame of image of the first image to obtain M feature points of the first image. It should be noted that, the previous frame image of the first image referred to herein refers to a first image obtained by preprocessing based on an original image obtained by the first camera at a previous time, that is, a previous frame first image. For example, when the first image A1 is obtained based on the first camera at the time i and the first image A2 is obtained based on the first camera at the time i+1, the second image A1 is considered to be the previous frame image of the first image A2. Considering that N feature points of the first image A1 are difficult to appear in the second image A2 when the robot is in a moving state, the number of feature points in the second image A2 obtained after optical flow tracking of the N feature points of the first image A1 is M, and M is N.
In some embodiments, it may occur that M is much smaller than N, e.gAt this time, the robot is considered to be likely to move in a larger range, and the currently obtained first image can be judged to be a key frame. In this case the robot will again supplement the first image with new feature points in such a way that FAST corner detection is performed on the first image, such that the feature points of the first image are up to N. For example, assuming that the i moment is the initial moment and N is 100, the robot will extract 100 corner points from the first image A1 as its feature points; further, it is assumed that only 40 feature points are obtained in the first image A2 by optical flow tracking of 100 feature points in the first image A1, and since 40 < 50, the first image A2 is judged to be a key frame; the robot extracts the corner points of the first image A2, supplements 60 corner points, and uses the corner points and 40 feature points obtained by optical flow tracking as final feature points of the first image A2.
Step 102, finding out at least one pair of matching line features in the first image and the second image.
In the embodiment of the application, the robot also searches at least one pair of matched line features in the first image and the second image, wherein each pair of line features comprises a first line feature and a second line feature, the first line feature is in the first image, and the second line feature is in the second image. That is, one line feature in the first image and a line feature in the second image that matches the line feature constitute a pair of line features. For example, the robot may perform a line feature extraction operation on the first image and the second image, respectively, to obtain at least one line feature (denoted as a third line feature) extracted in the first image, and at least one line feature (denoted as a fourth line feature) extracted in the second image; then, the robot may match each third line feature with each fourth line feature line, respectively, and obtain at least one pair of line features based on the matched third line features and fourth line features. Similar to step 101, no duplication of line features occurs in different pairs of line features.
For any image, the line feature extraction operation may be implemented as follows: the line segments in the image are extracted based on a preset straight line extraction algorithm, wherein the straight line extraction algorithm can be LSD (Line Segment Detector) algorithm or other algorithms, and the straight line extraction algorithm is not limited herein. In order to avoid the false extraction of the line feature and improve the subsequent processing efficiency of the line feature, only the line segments longer than the preset length are considered, for example, only the line segments with the pixel length exceeding 30 pixels can be considered. And under the influence of image noise or local blurring, one long line segment can be extracted into a plurality of short line segments in the extraction process, so that the robot can also detect whether two adjacent line segments are in the same straight line or not, and splice the two adjacent line segments in the same straight line, so that the continuity of line segment tracking is ensured. Wherein, two adjacent line segments refer to: the end point of one line segment is within a preset distance of the start point of the other line segment. It should be noted that, the end point and the start point of the line segment are determined by the straight line extraction algorithm, which is not described herein. After the above process is completed, the line segments in the obtained image are the line features of the image.
Illustratively, the robot may determine whether two adjacent line segments are in the same straight line by: respectively marking two adjacent line segments as a first line segment and a second line segment, and respectively calculating the normal vector of the first line segment and the normal vector of the second line segment; if the error of the normal vector of the first line segment and the normal vector of the second line segment is within a preset error range, determining that the first line segment and the second line segment are in the same straight line. Referring to fig. 2 and 3, fig. 2 shows an example of two adjacent line segments in the same line, and fig. 3 shows an example of a new line segment obtained by splicing two adjacent line segments in the same line shown in fig. 2. It can be seen that two adjacent line segments (line segment 1 and line segment 2) in the same straight line are spliced into a new line segment 3, and the new line segment 3 can be used as a line feature of the image.
In some embodiments, the robot may describe the extracted line features with LBD (Line Band Discriptor) descriptors and determine whether the two line features match based on LBD descriptors.
In some embodiments, the robot may have a mismatch when determining whether two line features match. The embodiment of the application can eliminate two line features which can be mismatched by the following modes: and respectively acquiring the midpoint and the slope of the two line features to be detected so as to judge whether the two line features are in mismatching.
For each pair of matched third and fourth line features, determining whether the third and fourth line features are mismatched may be based on a midpoint of the third line feature, a midpoint of the fourth line feature, a slope of the third line feature, and a slope of the fourth line feature, specifically: acquiring the coordinate of the midpoint of the third line feature in the first image, and recording the coordinate as a first coordinate; acquiring coordinates of the midpoint of the fourth line feature in the second image, and recording the coordinates as second coordinates; detecting whether the distance between the first coordinate and the second coordinate is within a preset error distance; calculating an inclination angle of the third line feature with respect to an x-axis of an image coordinate system based on a slope of the third line feature, and recording the inclination angle as a first angle; calculating an inclination angle of the fourth line feature with respect to an x-axis of an image coordinate system based on a slope of the fourth line feature, and recording the inclination angle as a second angle; detecting whether the angle difference value of the first angle and the second angle is within a preset error angle; the third line feature and the fourth line feature are considered to be exactly matched only if the distance between the first coordinate and the second coordinate is within a predetermined error distance and the angle difference between the first angle and the second angle is within a predetermined error angle, and may be determined as a pair of line features.
Similarly, the line features extracted from the first image may also be screened out once in the manner described above, specifically: based on the optical flow tracking mode, the matching relation between the line characteristics obtained by the previous frame image of the first image and the line characteristics obtained by the first image is determined. For convenience of explanation, in one matching relationship, a line feature obtained from a previous frame image of a first image is referred to as a fifth line feature, and a line feature obtained from the first image is referred to as a sixth line feature; since the robot generally does not move significantly in a short time, for each pair of the fifth line feature and the sixth line feature that are matched, it can be determined whether the fifth line feature and the sixth line feature are mismatched based on the midpoint of the fifth line feature, the midpoint of the sixth line feature, the slope of the fifth line feature, and the slope of the sixth line feature, specifically: acquiring coordinates of a midpoint of the fifth line feature in a previous frame of image of the first image, and marking the coordinates as third coordinates; acquiring coordinates of the midpoint of the sixth line feature in the first image, and recording the coordinates as fourth coordinates; detecting whether the distance between the third coordinate and the fourth coordinate is within a preset error distance; calculating the inclination angle of the fifth line feature relative to the x-axis of the image coordinate system based on the slope of the fifth line feature, and recording the inclination angle as a third angle; calculating the inclination angle of the sixth line feature relative to the x-axis of the image coordinate system based on the slope of the sixth line feature, and recording the inclination angle as a fourth angle; detecting whether the angle difference value between the third angle and the fourth angle is within a preset error angle; only if the distance between the third coordinate and the fourth coordinate is within a preset error distance, and the angle difference between the third angle and the fourth angle is within a preset error angle, determining that the fifth line feature and the sixth line feature are actually matched, and retaining the sixth line feature at the moment; otherwise, it may be determined that the fifth line feature and the sixth line feature are not matched, and then the sixth line feature may be removed from the first image, so as to implement screening of the line features extracted from the first image.
In some embodiments, considering that not all line features in the first image may find a matching line feature in the second image, i.e. there may be some line features in the first image that cannot find a matching line feature in the second image, the line features in the first image may be divided into two classes: the first class is line features that can be matched in the second image and the second class is line features that cannot be matched in the second image. For the first class, line features matching itself can be obtained directly based on the second image to obtain line feature pairs. For the second class, line features matched with the first camera can be acquired based on a previous frame image of the first image acquired by the first camera, so as to acquire line feature pairs. By the method, each line characteristic reserved after the first image screening can form a corresponding pair of line characteristics. For convenience of description, the first type of line feature may be referred to as a binocular line feature, and the second type of line feature may be referred to as a non-binocular line feature, so that in the case that the non-binocular line feature exists, the robot may find at least one pair of matched line features in the first image and the previous frame image of the first image based on the non-binocular line feature, where each pair of line features formed in this way includes one non-binocular first line feature and one non-binocular second line feature, the non-binocular first line feature is in the first image, and the non-binocular second line feature is in the previous frame image of the first image. As such, the pair of line features ultimately obtained by the robot may include not only at least one pair of line features obtained based on the binocular line features, the first image and the second image, but also at least one pair of line features obtained based on the non-binocular line features, the first image and a previous frame of the first image.
And 103, obtaining the predicted pose of the robot based on the at least one pair of line features, the at least one pair of feature points and inertial data output by an inertial measurement unit of the robot.
In the embodiment of the present application, the robot is preset with an objective function, and the objective function may be constrained based on at least one pair of feature points determined in step 101, at least one pair of line features determined in step 102, and inertial data output by an inertial measurement unit (Inertial Measurement Unit, IMU) of the robot, where the objective function specifically is:
wherein B represents a set of inertial data output by the IMU; c represents a set of at least one pair of feature points obtained in step 101; l represents a set of at least one pair of line features obtained in step 102; x represents a system state estimation value of the robot, and the system state estimation value comprises the pose of the robot and the pose of the characteristic points and the line characteristics in space; z represents a system state observation value of the robot; r represents the difference between the observed value of the system state and the estimated value of the system state, i.e. the system state difference.
It should be noted that the set L may include line feature pairs obtained based on non-binocular line features, the first image and a previous frame of the first image. For example, the robot extracted 10 line features in the first image; of the 10 line features, 1 line feature is removed because the matching line feature cannot be found in the previous frame image, and only 9 line features remain; that is, the 9 line features can find the matched line features in the previous frame image, and are marked as line features 1, 2, 3, 4, 5, 6, 7, 8 and 9; by matching with the line features extracted from the second image, if the line features 1, 3 and 4 cannot be matched with any line feature in the second image, the line features 1, 3 and 4 are respectively matched with the line features in the previous frame image to form 3 line features; the line features 2, 5, 6, 7, 8 and 9 will respectively form 6 pairs of line features together with the matched line features in the second image; thus, 9 pairs of line features are constructed, and the 9 pairs of line features can form a set L.
As can be seen from the above formula, r specifically includes three parts:
the first part is the residual of the integral value of the IMU and the true value.
The second part is the residual of the coordinates (i.e. pixel positions) obtained after the three-dimensional feature points are re-projected back to the two-dimensional image coordinate system and the coordinates of the feature points actually observed in the first image. Wherein, the three-dimensional characteristic points are obtained by binocular triangularization of a pair of characteristic points.
The third part is similar to the principle of the second part, and is a three-dimensional line characteristic re-projection residual. Referring to fig. 4, the three-dimensional line characteristic re-projection residual refers to: a three-dimensional line feature is a perpendicular distance between a straight line obtained by projecting a world coordinate system onto a normalization plane of the first camera and the second camera and a starting point and an ending point of an original line feature of the first image in the normalization plane. Wherein the three-dimensional line feature is obtained by triangulating a pair of line features.
These 3 residuals are added together to form the objective function. And (3) optimizing a system state estimated value X containing the pose of the robot by a least square method in a tight coupling way, so that the value of the whole objective function is minimum, wherein the X is an optimal solution, and the X of the optimal solution contains the pose predicted value of the current robot.
In some embodiments, considering that the three-dimensional feature points and line features are also optimization variables of the objective function, both are optimized together when iteratively solving the objective function; that is, the pose of the robot, the three-dimensional feature points, and the three-dimensional line features are iteratively adjusted together so that the objective function is minimized. Thus, the X of the optimal solution contains the three-dimensional feature points and line features after optimization besides the pose predicted value of the current robot. The robot can save the positions of the optimized characteristic points and line characteristics in the space and the characteristic description thereof to form a word bag, and the word bag is saved as a map for loop detection and correction; meanwhile, the map can also be used for repositioning in the next navigation positioning process.
From the above, according to the embodiment of the application, the pose of the robot is predicted by combining the characteristic points and the line characteristics, and the geometric structure information in the environment where the robot is located can be obtained by combining the characteristic points and the line characteristics, so that the robot can realize precise and robust pose prediction under challenging weak texture and low visible light scenes.
Corresponding to the pose prediction method provided by the embodiment of the application, the pose prediction device is applied to a robot provided with a binocular camera, and the binocular camera comprises a first camera and a second camera. Referring to fig. 5, a pose prediction apparatus 500 according to an embodiment of the present application includes:
A first searching unit 501, configured to search for at least one pair of matched feature points in a first image and a second image, where each pair of feature points includes a first feature point and a second feature point, the first image is obtained based on an image acquired at a current moment of the first camera, the second image is obtained based on an image acquired at a current moment of the second camera, the first feature point is in the first image, and the second feature point is in the second image;
a second searching unit 502, configured to search at least one pair of matched line features in the first image and the second image, where each pair of line features includes a first line feature and a second line feature, the first line feature is in the first image, and the second line feature is in the second image;
and a prediction unit 503 configured to obtain a predicted pose of the robot based on the at least one pair of line features, the at least one pair of feature points, and inertial data output from the inertial measurement unit of the robot.
Optionally, the first search unit 501 includes:
a feature point extraction subunit, configured to extract at least one feature point from the first image;
And the characteristic point matching subunit is used for respectively searching the characteristic points matched with each characteristic point in the first image in the second image so as to obtain at least one pair of characteristic points.
Optionally, the feature point extracting subunit includes:
the detection subunit is used for detecting whether the first image is obtained based on a first frame image acquired after the first camera is started;
a first feature point extraction subunit, configured to extract feature points based on a preset first feature point extraction manner if the first image is obtained based on the first frame image, where the first feature point extraction manner is related to a corner point;
and a second feature point extraction subunit, configured to extract feature points based on a preset second feature point extraction method if the first image is not obtained based on the first frame image, where the second feature point extraction method is related to optical flow.
Optionally, the second search unit 502 includes:
a line feature extraction subunit, configured to perform a line feature extraction operation on the first image and the second image, respectively, to obtain a third line feature and a fourth line feature, where the third line feature is a line feature extracted from the first image, and the fourth line feature is a line feature extracted from the second image;
A line feature matching subunit, configured to match each third line feature with each fourth line feature line respectively;
and the line characteristic pair acquisition subunit is used for acquiring at least one pair of line characteristics based on the matched third line characteristic and fourth line characteristic.
Optionally, the line feature extraction subunit includes:
a line segment extraction subunit, configured to extract, for any one of the first image and the second image, a line segment exceeding a preset length in the image based on a preset straight line extraction algorithm;
the line segment splicing subunit is configured to splice two adjacent line segments that are in the same straight line, where the two adjacent line segments refer to: the end point of one line segment is within a preset distance of the start point of the other line segment.
Optionally, the line feature pair acquiring subunit includes:
a judging subunit, configured to judge, for each pair of the matched third line feature and fourth line feature, whether the third line feature and the fourth line feature are mismatched based on a midpoint of the third line feature, a midpoint of the fourth line feature, a slope of the third line feature, and a slope of the fourth line feature;
And a determining subunit configured to determine the third line feature and the fourth line feature as a pair of line features if the third line feature and the fourth line feature are not mismatched.
Optionally, the prediction unit 503 includes:
an objective function optimizing subunit, configured to optimize an objective function based on a preset optimizing method, where the objective function uses the at least one pair of line features, the at least one pair of feature points, and inertial data output by an inertial measurement unit of the robot as constraint conditions;
and the predicted pose acquisition subunit is used for acquiring the predicted pose of the robot based on the optimized objective function.
From the above, according to the embodiment of the application, the pose of the robot is predicted by combining the characteristic points and the line characteristics, and the geometric structure information in the environment where the robot is located can be obtained by combining the characteristic points and the line characteristics, so that the robot can realize precise and robust pose prediction under challenging weak texture and low visible light scenes.
Referring to fig. 6, the robot 6 in the embodiment of the present application includes: memory 601, one or more processors 602 (only one shown in fig. 6), binocular camera 603, and computer programs stored on memory 601 and executable on the processors. The binocular camera 603 includes a first camera and a second camera; the memory 601 is used for storing software programs and units, and the processor 602 executes various functional applications and data processing by running the software programs and units stored in the memory 601 to acquire resources corresponding to the preset events. Specifically, the processor 602 implements the following steps by running the above-described computer program stored in the memory 601:
At least one pair of matched characteristic points are found out from a first image and a second image, wherein each pair of characteristic points comprises a first characteristic point and a second characteristic point, the first image is obtained based on an image acquired at the current moment of the first camera, the second image is obtained based on an image acquired at the current moment of the second camera, the first characteristic point is in the first image, and the second characteristic point is in the second image;
finding out at least one pair of matched line features in the first image and the second image, wherein each pair of line features comprises a first line feature and a second line feature, the first line feature is in the first image, and the second line feature is in the second image;
and obtaining a predicted pose of the robot based on the at least one pair of line features, the at least one pair of feature points, and inertial data output by an inertial measurement unit of the robot.
In a second possible implementation manner provided by the first possible implementation manner, assuming that the foregoing is a first possible implementation manner, the searching at least one pair of matched feature points in the first image and the second image includes:
Extracting at least one feature point from the first image;
and searching for the characteristic points matched with the characteristic points in the first image in the second image respectively to obtain at least one pair of characteristic points.
In a third possible implementation manner provided by the second possible implementation manner, the extracting at least one feature point from the first image includes:
detecting whether the first image is obtained based on a first frame image acquired after the first camera is started;
if the first image is obtained based on the first frame image, extracting feature points based on a preset first feature point extraction mode, wherein the first feature point extraction mode is related to corner points;
if the first image is not obtained based on the first frame image, extracting feature points based on a preset second feature point extraction mode, wherein the second feature point extraction mode is related to optical flow.
In a fourth possible implementation manner provided by the first possible implementation manner, the searching for at least one pair of matching line features in the first image and the second image includes:
Performing line feature extraction operation on the first image and the second image respectively to obtain a third line feature and a fourth line feature, wherein the third line feature is the line feature extracted from the first image, and the fourth line feature is the line feature extracted from the second image;
matching each third line characteristic with each fourth line characteristic row respectively;
at least one pair of line features is obtained based on the matched third and fourth line features.
In a fifth possible implementation manner provided by the fourth possible implementation manner, the line feature extraction operation includes, for any one of the first image and the second image:
extracting line segments exceeding a preset length from the image based on a preset straight line extraction algorithm;
splicing two adjacent line segments in the same straight line, wherein the two adjacent line segments refer to: the end point of one line segment is within a preset distance of the start point of the other line segment.
In a sixth possible implementation manner provided by the fourth possible implementation manner, the obtaining at least one pair of line features based on the matched third line feature and fourth line feature includes:
For each pair of matched third and fourth line features, determining whether the third and fourth line features are mismatched based on a midpoint of the third line feature, a midpoint of the fourth line feature, a slope of the third line feature, and a slope of the fourth line feature;
and if the third line feature and the fourth line feature are not mismatched, determining the third line feature and the fourth line feature as a pair of line features.
In a seventh possible embodiment provided by the first possible embodiment, the predicting the pose of the robot based on the at least one pair of line features, the at least one pair of feature points, and inertial data output from an inertial measurement unit of the robot includes:
optimizing an objective function based on a preset optimization method, wherein the objective function adopts the at least one pair of line characteristics, the at least one pair of characteristic points and inertial data output by an inertial measurement unit of the robot as constraint conditions;
and obtaining the predicted pose of the robot based on the optimized objective function.
It should be appreciated that in embodiments of the present application, the processor 602 may be a central processing unit (Central Processing Unit, CPU), which may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSPs), application specific integrated circuits (Application Specific Integrated Circuit, ASICs), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
Memory 601 may include read only memory and random access memory and provides instructions and data to processor 602. Some or all of the memory 601 may also include non-volatile random access memory. For example, the memory 601 may also store information of a device class.
From the above, according to the embodiment of the application, the pose of the robot is predicted by combining the characteristic points and the line characteristics, and the geometric structure information in the environment where the robot is located can be obtained by combining the characteristic points and the line characteristics, so that the robot can realize precise and robust pose prediction under challenging weak texture and low visible light scenes.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions. The functional units and modules in the embodiment may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of a software functional unit. In addition, the specific names of the functional units and modules are only for distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working process of the units and modules in the above system may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and in part, not described or illustrated in any particular embodiment, reference is made to the related descriptions of other embodiments.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of external device software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the system embodiments described above are merely illustrative, e.g., the division of modules or units described above is merely a logical functional division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection via interfaces, devices or units, which may be in electrical, mechanical or other forms.
The units described above as separate components may or may not be physically separate, and components shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
The integrated units described above, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the present application may also be implemented by implementing all or part of the flow of the method of the above embodiment, or by instructing the associated hardware by a computer program, where the computer program may be stored on a computer readable storage medium, and where the computer program, when executed by a processor, may implement the steps of each of the method embodiments described above. The computer program comprises computer program code, and the computer program code can be in a source code form, an object code form, an executable file or some intermediate form and the like. The above computer readable storage medium may include: any entity or device capable of carrying the computer program code described above, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer readable Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier wave signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the content of the computer readable storage medium described above may be appropriately increased or decreased according to the requirements of the jurisdiction's legislation and the patent practice, for example, in some jurisdictions, the computer readable storage medium does not include electrical carrier signals and telecommunication signals according to the legislation and the patent practice.
The above embodiments are only for illustrating the technical solution of the present application, and are not limiting; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application, and are intended to be included in the scope of the present application.
Claims (8)
1. The pose prediction method is characterized by being applied to a robot provided with a binocular camera, wherein the binocular camera comprises a first camera and a second camera, and the pose prediction method comprises the following steps:
at least one pair of matched characteristic points are found out from a first image and a second image, wherein each pair of characteristic points comprises a first characteristic point and a second characteristic point, the first image is obtained based on an image collected at the current moment of the first camera, the second image is obtained based on an image collected at the current moment of the second camera, the first characteristic point is in the first image, and the second characteristic point is in the second image;
Finding out at least one pair of matched line features in the first image and the second image, wherein each pair of line features comprises a first line feature and a second line feature, the first line feature is in the first image, and the second line feature is in the second image;
based on the at least one pair of line features, the at least one pair of feature points and inertial data output by an inertial measurement unit of the robot, obtaining a predicted pose of the robot;
and finding out at least one pair of matched characteristic points in the first image and the second image, wherein the method comprises the following steps:
extracting at least one feature point from the first image;
searching for feature points matched with each feature point in the first image in the second image respectively to obtain at least one pair of feature points;
wherein the extracting at least one feature point from the first image includes:
detecting whether the first image is obtained based on a first frame image acquired after the first camera is started;
if the first image is obtained based on the first frame image, extracting N feature points based on a preset first feature point extraction mode, wherein the first feature point extraction mode is related to corner points, and N is a preset feature point number threshold;
If the first image is not obtained based on the first frame image, extracting feature points based on a preset second feature point extraction mode, wherein the second feature point extraction mode is related to optical flow;
the number of the feature points extracted based on the second feature point extraction mode is less than or equal toIn the above, the feature points are extracted again based on the first feature point extraction means so that the number of feature points of the first image is supplemented to N.
2. The pose prediction method according to claim 1, wherein the finding at least one pair of matching line features in the first image and the second image comprises:
performing line feature extraction operation on the first image and the second image respectively to obtain a third line feature and a fourth line feature, wherein the third line feature is the line feature extracted from the first image, and the fourth line feature is the line feature extracted from the second image;
matching each third line characteristic with each fourth line characteristic row respectively;
at least one pair of line features is obtained based on the matched third and fourth line features.
3. The pose prediction method according to claim 2, wherein the line feature extraction operation includes, for any one of the first image and the second image:
extracting line segments exceeding a preset length from the image based on a preset straight line extraction algorithm;
splicing two adjacent line segments in the same straight line, wherein the two adjacent line segments refer to: the end point of one line segment is within a preset distance of the start point of the other line segment.
4. The pose prediction method according to claim 2, wherein the obtaining at least one pair of line features based on the matched third line feature and fourth line feature comprises:
for each pair of matched third and fourth line features, determining whether the third and fourth line features are mismatched based on a midpoint of the third line feature, a midpoint of the fourth line feature, a slope of the third line feature, and a slope of the fourth line feature;
if the third line feature and the fourth line feature are not mismatched, the third line feature and the fourth line feature are determined to be a pair of line features.
5. The pose prediction method according to claim 1, wherein predicting the pose of the robot based on the at least one pair of line features, the at least one pair of feature points, and inertial data output by an inertial measurement unit of the robot, comprises:
Optimizing an objective function based on a preset optimization method, wherein the objective function adopts the at least one pair of line characteristics, the at least one pair of characteristic points and inertial data output by an inertial measurement unit of the robot as constraint conditions;
and obtaining the predicted pose of the robot based on the optimized objective function.
6. The utility model provides a pose prediction device, its characterized in that, pose prediction device is applied to the robot that is provided with binocular camera, binocular camera includes first camera and second camera, pose prediction device includes:
the first searching unit is used for searching at least one pair of matched characteristic points in a first image and a second image, wherein each pair of characteristic points comprises a first characteristic point and a second characteristic point, the first image is obtained based on an image acquired at the current moment of the first camera, the second image is obtained based on an image acquired at the current moment of the second camera, the first characteristic point is in the first image, and the second characteristic point is in the second image;
the second searching unit is used for searching at least one pair of matched line features in the first image and the second image, wherein each pair of line features comprises a first line feature and a second line feature, the first line feature is in the first image, and the second line feature is in the second image;
A prediction unit, configured to obtain a predicted pose of the robot based on the at least one pair of line features, the at least one pair of feature points, and inertial data output by the inertial measurement unit of the robot;
wherein the first search unit includes:
a feature point extraction subunit, configured to extract at least one feature point from the first image;
the characteristic point matching subunit is used for respectively searching the characteristic points matched with each characteristic point in the first image in the second image so as to obtain at least one pair of characteristic points;
wherein the feature point extraction subunit includes:
the detection subunit is used for detecting whether the first image is obtained based on a first frame image acquired after the first camera is started;
a first feature point extraction subunit, configured to extract N feature points based on a preset first feature point extraction manner if the first image is obtained based on the first frame image, where the first feature point extraction manner is related to a corner point, and N is a preset feature point number threshold;
a second feature point extraction subunit, configured to extract feature points based on a preset second feature point extraction method if the first image is not obtained based on the first frame image, where the number of feature points extracted based on the second feature point extraction method is equal to or less And extracting feature points again based on the first feature point extraction means such that the number of feature points of the first image is supplemented to N, wherein the second feature point extraction means is related to optical flow.
7. A robot comprising a memory, a processor, a binocular camera, and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the method of any one of claims 1 to 5 when executing the computer program.
8. A computer readable storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the method according to any one of claims 1 to 5.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110194534.2A CN112950709B (en) | 2021-02-21 | 2021-02-21 | Pose prediction method, pose prediction device and robot |
| PCT/CN2021/124611 WO2022174603A1 (en) | 2021-02-21 | 2021-10-19 | Pose prediction method, pose prediction apparatus, and robot |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110194534.2A CN112950709B (en) | 2021-02-21 | 2021-02-21 | Pose prediction method, pose prediction device and robot |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112950709A CN112950709A (en) | 2021-06-11 |
| CN112950709B true CN112950709B (en) | 2023-10-24 |
Family
ID=76244975
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110194534.2A Active CN112950709B (en) | 2021-02-21 | 2021-02-21 | Pose prediction method, pose prediction device and robot |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN112950709B (en) |
| WO (1) | WO2022174603A1 (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112950709B (en) * | 2021-02-21 | 2023-10-24 | 深圳市优必选科技股份有限公司 | Pose prediction method, pose prediction device and robot |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109166149A (en) * | 2018-08-13 | 2019-01-08 | 武汉大学 | A kind of positioning and three-dimensional wire-frame method for reconstructing and system of fusion binocular camera and IMU |
| CN109579840A (en) * | 2018-10-25 | 2019-04-05 | 中国科学院上海微系统与信息技术研究所 | A kind of close coupling binocular vision inertia SLAM method of dotted line Fusion Features |
| CN110060277A (en) * | 2019-04-30 | 2019-07-26 | 哈尔滨理工大学 | A kind of vision SLAM method of multiple features fusion |
| CN111160298A (en) * | 2019-12-31 | 2020-05-15 | 深圳市优必选科技股份有限公司 | Robot and pose estimation method and device thereof |
| EP3680809A1 (en) * | 2017-09-22 | 2020-07-15 | Huawei Technologies Co., Ltd. | Visual slam method and apparatus based on point and line characteristic |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107747941B (en) * | 2017-09-29 | 2020-05-15 | 歌尔股份有限公司 | Binocular vision positioning method, device and system |
| CN110763251B (en) * | 2019-10-18 | 2021-07-13 | 华东交通大学 | Method and system for optimizing visual inertial odometer |
| CN112115980A (en) * | 2020-08-25 | 2020-12-22 | 西北工业大学 | Binocular vision odometer design method based on optical flow tracking and point line feature matching |
| CN112950709B (en) * | 2021-02-21 | 2023-10-24 | 深圳市优必选科技股份有限公司 | Pose prediction method, pose prediction device and robot |
-
2021
- 2021-02-21 CN CN202110194534.2A patent/CN112950709B/en active Active
- 2021-10-19 WO PCT/CN2021/124611 patent/WO2022174603A1/en active Application Filing
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3680809A1 (en) * | 2017-09-22 | 2020-07-15 | Huawei Technologies Co., Ltd. | Visual slam method and apparatus based on point and line characteristic |
| CN109166149A (en) * | 2018-08-13 | 2019-01-08 | 武汉大学 | A kind of positioning and three-dimensional wire-frame method for reconstructing and system of fusion binocular camera and IMU |
| CN109579840A (en) * | 2018-10-25 | 2019-04-05 | 中国科学院上海微系统与信息技术研究所 | A kind of close coupling binocular vision inertia SLAM method of dotted line Fusion Features |
| CN110060277A (en) * | 2019-04-30 | 2019-07-26 | 哈尔滨理工大学 | A kind of vision SLAM method of multiple features fusion |
| CN111160298A (en) * | 2019-12-31 | 2020-05-15 | 深圳市优必选科技股份有限公司 | Robot and pose estimation method and device thereof |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2022174603A1 (en) | 2022-08-25 |
| CN112950709A (en) | 2021-06-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111354042B (en) | Feature extraction method and device of robot visual image, robot and medium | |
| CN110246147B (en) | Visual inertial odometer method, visual inertial odometer device and mobile equipment | |
| CN110427917B (en) | Method and device for detecting key points | |
| EP2491529B1 (en) | Providing a descriptor for at least one feature of an image | |
| US7554575B2 (en) | Fast imaging system calibration | |
| WO2018120038A1 (en) | Method and device for target detection | |
| US20190011263A1 (en) | Method and apparatus for determining spacecraft attitude by tracking stars | |
| CN109640066B (en) | Method and device for generating high-precision dense depth image | |
| EP2751742A1 (en) | Method of matching image features with reference features | |
| CN110349212B (en) | Optimization method and device for instant positioning and map construction, medium and electronic equipment | |
| US10460471B2 (en) | Camera pose estimating method and system | |
| CN101834986A (en) | Imaging device, mobile body detecting method, mobile body detecting circuit and program | |
| CN111161347A (en) | Method and equipment for initializing SLAM | |
| CN109447902B (en) | Image stitching method, device, storage medium and equipment | |
| CN111461998A (en) | Environment reconstruction method and device | |
| CN112001883B (en) | Optimization method and device for vehicle target image and computer equipment | |
| El Bouazzaoui et al. | Enhancing RGB-D SLAM performances considering sensor specifications for indoor localization | |
| CN112950709B (en) | Pose prediction method, pose prediction device and robot | |
| Remondino et al. | Evaluating hand-crafted and learning-based features for photogrammetric applications | |
| CN107113421B (en) | The detection method and device of a kind of optical system imaging quality | |
| CN105574844B (en) | Rdaiation response Function Estimation method and apparatus | |
| CN112270748B (en) | Three-dimensional reconstruction method and device based on image | |
| CN113012084A (en) | Unmanned aerial vehicle image real-time splicing method and device and terminal equipment | |
| CN112288817B (en) | Three-dimensional reconstruction processing method and device based on image | |
| US9842402B1 (en) | Detecting foreground regions in panoramic video frames |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |