CN110647609A - Visual map positioning method and system - Google Patents
Visual map positioning method and system Download PDFInfo
- Publication number
- CN110647609A CN110647609A CN201910877005.5A CN201910877005A CN110647609A CN 110647609 A CN110647609 A CN 110647609A CN 201910877005 A CN201910877005 A CN 201910877005A CN 110647609 A CN110647609 A CN 110647609A
- Authority
- CN
- China
- Prior art keywords
- map
- relative
- picture
- visual
- transformation matrix
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/29—Geographical information databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/587—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using geographical or spatial information, e.g. location
Abstract
The invention discloses a visual map positioning method and a system, wherein the visual map positioning method comprises the following steps: calculating a transformation matrix of the local map relative to the global map by matching descriptors of the local map 3d point and the global map 3d point; acquiring the position and orientation of the set photo relative to the reference photo; and transforming the position and orientation of the acquired setting photo relative to the reference photo into a global map by using the transformation matrix. The visual map positioning method and the visual map positioning system can obtain the estimated value of the current camera position with very low delay, and update the camera position to obtain a more accurate position after the matching of 3d to 2d with longer operation time is finished, so that the positioning delay is reduced.
Description
Technical Field
The invention belongs to the technical field of visual maps, relates to a positioning method, and particularly relates to a visual map positioning method and a visual map positioning system.
Background
After matching the picture with the visual map, the position and orientation of the camera taking the picture can be obtained. A common form of visual positioning map is a set of 3d points with descriptors. The visual positioning map stores information of a plurality of 3d points, and the information comprises the positions of the 3d points and a plurality of descriptors. A descriptor is a piece of data, such as 32 bytes or 32 floats. The descriptor represents the feature of the corresponding feature point of this 3d point in the photograph.
Calculation of the visual descriptor, see fig. 1, each arrow in the right drawing may be represented by a vector, and the combination of vectors represented by all arrows is a descriptor of a feature point.
Visual map-based positioning refers to calculating the position and orientation of a photograph relative to a map. For a photo, some corner points can be extracted, and each corner point can also extract a descriptor; these descriptors store the pattern features near this corner point. While these descriptors can be matched with the descriptors of the 3d points in the map. That is, it is found which points in the map, 3d point and the photograph, 2d point are the same point through the difference of the descriptors.
By matching the descriptors, the corresponding relation from the 2d point of many images to the 3d point of the positioning map can be established. Using this correspondence, the shooting position and orientation of each photograph can be calculated.
However, the 3 d-to-2 d matching is generally more computationally intensive, because the number of 3d points in the map is much larger than the number of feature points of a 2d point. The method of calculating the camera position and orientation by 3d to 2d matching cannot be performed with a too high frequency. In addition, some areas are not covered by the map, and the areas cannot be located by the method.
The solution to this situation is to use a visual odometer which can calculate the position and orientation of successive pictures with respect to the previous picture. And the visual odometer only needs to match pictures, and the speed is faster than the speed of matching 3d to 2d points. Let it be assumed that the matrix T _ w _ i represents the position and orientation of the previous picture. t _ i _ j is the position of the next picture relative to the previous picture as calculated by the visual odometer. Then the position of the next picture relative to the map can be obtained by T _ w _ j ═ T _ w _ i ═ T _ i _ j; namely, the position of the latter photo relative to the map can be obtained without performing the matching task of 3d to 2d points.
Each triangle in fig. 2 represents the position and orientation of the camera at a different time, and a successful match with the map at time t4 yields a position that differs from the position that was extrapolated using visual odometry. A correction is made to the accumulated error of the visual odometer at time t4 by a match of 3d to 2 d.
However, this has the disadvantage that the position of the previous picture relative to the map must be obtained when calculating the position of the next picture. If the time for the 3d to 2d match is long, such as taking 1 second to get the result, the entire positioning system will have a long delay.
The existing visual map positioning mode is mainly based on a Kalman filtering method, and has the following defects:
(1) without using information between image sequences, the localization quality is poor in areas without map coverage. However, the method fuses the information between the image sequences obtained by the visual odometer and the matching information of the images and the map together, and can obtain a high-precision positioning result through the information of the odometer in the area with poor map quality.
(2) Generally, matching of pictures and maps takes time, so that matching of pictures and maps is not performed every picture. The picture position which is not matched with the map in the Kalman filtering-based method can be quickly obtained through the prediction of a filter. But there is a large delay in the positioning results for which map matching is required. For example, the matching time of the picture and the map is 1s, the positioning delay of the picture is 1 s.
In view of the above, there is an urgent need to design a visual map positioning method to overcome the above-mentioned shortcomings of the existing visual map positioning methods.
Disclosure of Invention
The invention provides a visual map positioning method and a visual map positioning system, which can obtain an estimated value of the current camera position with very low delay, and update the camera position to obtain a more accurate position after the matching of 3d to 2d with longer operation time is finished, thereby reducing the positioning delay.
In order to solve the technical problem, according to one aspect of the present invention, the following technical solutions are adopted:
a visual map positioning method, the visual map positioning method comprising:
calculating a transformation matrix of the local map relative to the global map by matching descriptors of the local map 3d point and the global map 3d point;
acquiring the position and orientation of the set photo relative to the reference photo;
and transforming the position and orientation of the acquired setting photo relative to the reference photo into a global map by using the transformation matrix.
As an embodiment of the present invention, the process of calculating the transformation matrix of the local map with respect to the global map includes:
s1, acquiring a reference photo and a photo to be matched, and extracting feature points and descriptors of the reference photo and the photo to be matched;
step S2, calculating the position and orientation of the feature point of the picture to be matched relative to the feature point of the reference picture; the 3d points with known positions form a local map;
step S3, finding out the matching relationship between the local map and the global map by matching the descriptors from the local map 3d point to the global map 3d point;
and step S4, calculating a transformation matrix from the local map points to the global map.
As an embodiment of the present invention, in step S4, the global transformation is composed of a rotation amount R and a translation amount t; the following linear equations are listed: p2 ═ R × p1+ t;
listing a corresponding number of linear equations according to the logarithm of (p1, p 2); the unknowns R and t are obtained by solving the solution of the system of linear equations.
As an embodiment of the present invention, the process of acquiring the position and orientation of the setting photograph with respect to the reference photograph includes the step S5: the position and orientation of the corresponding picture relative to the reference photograph are calculated using a visual odometer.
As an embodiment of the present invention, the process of transforming the position and orientation of the acquired setting photo with respect to the reference photo into the global map includes:
step S6, obtaining the position of the corresponding picture relative to the map by using the transformation matrix;
and step S7, after receiving the subsequent pictures, obtaining the position of the current camera relative to the reference picture through the visual odometer, and then obtaining the position of the current camera relative to the map.
As an embodiment of the present invention, the process of transforming the position and orientation of the acquired setting photo relative to the reference photo into the global map further includes step S8, the subsequent pictures and the existing pictures together establishing more 3d points of the local map; and after the 3d point of the local map is matched with the global map, a more accurate transformation matrix can be obtained.
As an embodiment of the present invention, in step S1, for each descriptor of the reference photo, the most similar one descriptor is found in the photos to be matched; extracting N descriptors from the reference picture to obtain the matching of N pairs of feature points; and according to the difference sorting of the descriptors, keeping the set match with the minimum difference.
In step S1, as an embodiment of the present invention, the feature point is a place where the brightness change is severe in the picture, the FAST corner point is used as the feature point, the descriptor represents the visual feature around the feature point, and the ORB descriptor is used.
As an embodiment of the present invention, the flow of calculating the position of the 3d point in step S2 includes:
s21, obtaining a transformation matrix of the picture to be matched relative to the reference picture by using a five-point method;
step S22, the position of the 3d point relative to the reference picture is obtained using the triangulation method.
As an embodiment of the present invention, in step S3, the matching method includes:
step S31, descriptor matching: for each descriptor of the reference photo, finding the most similar one in the photos to be matched; extracting N descriptors from the reference picture to obtain the matching of N pairs of feature points; according to the difference ordering of the descriptors, keeping M matching pairs with the minimum difference;
and step S32, screening out wrong pairings in the pairings by using a RANSAC algorithm to obtain a final pairing.
As an embodiment of the present invention, in step S5, the position t _1_3 and orientation R _1_3 of the third picture relative to the first picture are calculated using a visual odometer;
the transformation matrix of the local map relative to the global map is marked as T _ w _ 1; in step S6, the transformation matrix T _ w _1 is used to obtain the position T _ w _3 of the third picture relative to the map, i.e., T _ w _1_ T _1_ 3.
As an embodiment of the present invention, in step S7, when a subsequent photo is received, the position t _1_ n of the current camera relative to the reference photo is obtained by the visual odometer; and obtaining the position of the current camera relative to the map through T _ w _ n-T _ w _ 1T _1_ n.
According to another aspect of the invention, the following technical scheme is adopted: a visual map positioning system, the visual map positioning system comprising:
the transformation matrix acquisition module is used for calculating a transformation matrix of the local map relative to the global map through descriptor matching of the local map 3d point and the global map 3d point;
the relative position and orientation acquisition module is used for acquiring the position and orientation of the set photo relative to the reference photo;
and the data transformation module is used for transforming the position and the orientation obtained by the visual odometer into a global map by using the transformation matrix.
As an embodiment of the present invention, the transformation matrix obtaining module includes:
a feature and description extraction unit for acquiring the reference picture and the picture to be matched, and extracting feature points and descriptors of the reference picture and the picture to be matched;
a position and orientation acquisition unit for calculating the position and orientation of the feature points of the picture to be matched with respect to the feature points of the reference picture; the 3d points with known positions form a local map;
a matching relationship obtaining unit, configured to find a matching relationship between the local map and the global map through descriptor matching from the local map 3d point to the global map 3d point;
-a transformation matrix calculation unit to calculate a transformation matrix from the local map points to the global map.
As an embodiment of the present invention, the transformation matrix calculation unit calculates the transformation matrix by:
the global transformation consists of a rotation amount R and a translation amount t; the following linear equations are listed: p2 ═ R × p1+ t;
listing a corresponding number of linear equations according to the logarithm of (p1, p 2); the unknowns R and t are obtained by solving the solution of the system of linear equations.
As an embodiment of the present invention, the relative position and orientation acquisition module includes a visual odometer; the data transformation module includes:
-a relative map position obtaining unit to obtain a position of the corresponding picture relative to the map using the transformation matrix;
a camera relative position acquisition unit for obtaining, by means of a visual odometer, the position of the current camera relative to the reference photograph and then to the map, after receiving the subsequent pictures.
As an embodiment of the present invention, for each descriptor of a reference photo, the feature and description extraction unit finds the most similar one of the descriptors in the photos to be matched; extracting N descriptors from the reference picture to obtain the matching of N pairs of feature points; and according to the difference sorting of the descriptors, keeping the set match with the minimum difference.
As an embodiment of the present invention, the position and orientation acquiring unit obtains a transformation matrix of the picture to be matched with respect to the reference picture by using a five-point method, and obtains the position of the 3d point with respect to the reference picture by using a triangulation method.
As an embodiment of the present invention, the matching relation obtaining unit includes:
-a descriptor matching subunit to find, for each descriptor of a reference photograph, the most similar one of the descriptors in the photographs to be matched; extracting N descriptors from the reference picture to obtain the matching of N pairs of feature points; according to the difference ordering of the descriptors, keeping M matching pairs with the minimum difference;
a pairing subunit, which screens out the wrong pairings among these pairings using the RANSAC algorithm, resulting in the final pairing.
As an embodiment of the present invention, the relative position and orientation obtaining module calculates a position t _1_3 and an orientation R _1_3 of the third picture relative to the first picture; the transformation matrix of the local map relative to the global map is marked as T _ w _ 1;
the relative map position obtaining unit obtains a position T _ w _3 of the third picture relative map by using the transformation matrix T _ w _1, namely T _ w _1, T _1_ 3.
As an embodiment of the present invention, the camera relative position acquisition unit acquires a position T _ w _ n ═ T _ w _1 × -T _1_ n of the current camera relative map.
The invention has the beneficial effects that: the visual map positioning method and the visual map positioning system can obtain the estimated value of the current camera position with very low delay, and update the camera position to obtain a more accurate position after the matching of 3d to 2d with longer operation time is finished, so that the positioning delay is reduced.
In the invention, the visual odometer does not depend on the result of map matching, so the positions of all pictures can be quickly obtained through the visual odometer. And after the parallel map matching is finished, updating the position of the latest picture by using the map matching result. Compared with a Kalman filtering-based method, the method has better parallelism. The invention also provides possibility for a map matching method with better use performance but lower speed in the future.
Drawings
Fig. 1 is a schematic diagram illustrating 3d information in a conventional visual positioning map.
Fig. 2 is a schematic diagram of positioning by using a visual odometer in a conventional visual positioning manner.
FIG. 3 is a flowchart illustrating a visual map positioning method according to an embodiment of the present invention.
FIG. 4 is a flowchart illustrating a visual map positioning method according to an embodiment of the present invention.
FIG. 5 is a flowchart illustrating a visual map positioning method according to an embodiment of the present invention.
Fig. 6 is a schematic diagram of acquiring a local map according to an embodiment of the present invention.
FIG. 7 is a diagram illustrating the calculation of a transformation matrix from a local map point to a global map according to an embodiment of the present invention.
Fig. 8 is a schematic diagram illustrating a local map and a global map being paired according to an embodiment of the present invention.
FIG. 9 is a block diagram of a visual map positioning system according to an embodiment of the present invention.
FIG. 10 is a block diagram of a visual map positioning system according to an embodiment of the present invention.
Detailed Description
Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
For a further understanding of the invention, reference will now be made to the preferred embodiments of the invention by way of example, and it is to be understood that the description is intended to further illustrate features and advantages of the invention, and not to limit the scope of the claims.
The description in this section is for several exemplary embodiments only, and the present invention is not limited only to the scope of the embodiments described. It is within the scope of the present disclosure and protection that the same or similar prior art means and some features of the embodiments may be interchanged.
The invention discloses a visual map positioning method, which comprises the following steps: calculating a transformation matrix of the local map relative to the global map by matching descriptors of the local map 3d point and the global map 3d point; acquiring the position and orientation of the set photo relative to the reference photo; and transforming the position and orientation of the acquired setting photo relative to the reference photo into a global map by using the transformation matrix.
FIG. 3 is a flowchart of a visual map positioning method according to an embodiment of the present invention; referring to fig. 3, in an embodiment of the present invention, the method for positioning a visual map includes:
step one, calculating a transformation matrix of a local map relative to a global map through descriptor matching of a local map 3d point and a global map 3d point;
step two, acquiring the position and the orientation of the set photo relative to the reference photo;
and thirdly, transforming the position and the orientation of the acquired set photo relative to the reference photo into a global map by using the transformation matrix.
Wherein, the step one and the step two have no requirement of the sequence and have no direct influence on each other.
FIG. 4 is a flowchart of a visual map positioning method according to an embodiment of the present invention; referring to fig. 4, in an embodiment of the present invention, the first step includes:
s1, acquiring a reference photo and a photo to be matched, and extracting feature points and descriptors of the reference photo and the photo to be matched;
step S2, calculating the position and orientation of the feature point of the picture to be matched relative to the feature point of the reference picture; the 3d points with known positions form a local map;
step S3, finding out the matching relationship between the local map and the global map by matching the descriptors from the local map 3d point to the global map 3d point;
and step S4, calculating a transformation matrix from the local map points to the global map.
In an embodiment of the present invention, in step S4, it is set that the global transformation is composed of a rotation amount R and a translation amount t (in an embodiment of the present invention, the global transformation includes others); the following linear equations are listed: p2 ═ R × p1+ t; since there are many pairs (p1, p2), the corresponding number of linear equations is listed according to the logarithm of (p1, p 2); the unknowns R and t are obtained by solving the solution of the system of linear equations.
With continued reference to fig. 4, in an embodiment of the present invention, the second step includes step S5 of calculating the position and orientation of the corresponding picture relative to the reference picture by using a visual odometer.
Referring to fig. 4, in an embodiment of the present invention, the second step includes:
step S6, obtaining the position of the corresponding picture relative to the map by using the transformation matrix;
and step S7, after receiving the subsequent pictures, obtaining the position of the current camera relative to the reference picture through the visual odometer, and then obtaining the position of the current camera relative to the map.
In an embodiment of the present invention, the second step further includes step S8, where the subsequent picture and the existing picture together establish more 3d points of the local map; and after the 3d point of the local map is matched with the global map, a more accurate transformation matrix can be obtained.
The visual positioning method provided by the invention can simultaneously carry out matching of the visual odometer and 3d to 2 d; that is, the present invention can obtain the position of the next camera with respect to the map based on only the result of the low-delay visual odometer even if the position of the previous image with respect to the map has not been calculated.
The 3d to 2d matching is also called matching of a local map and a global map. A map composed of feature points of known 3d positions resulting from matching between successive pictures is called a local map. The localization map is also called a global map.
The principle of the invention is that a transformation matrix of a local map relative to a global map is calculated, and then the position and the orientation obtained by the visual odometer are transformed into the global map by using the matrix. The existing method calculates the position of a certain photo based on the matching of 3d to 2d, and the method provided by the invention calculates the transformation of a local map relative to a global map through the matching of a 3d point of the local map and a 3d point of the global map. Because the position and orientation of the previous picture relative to the map are not relied on, the visual odometer can calculate the current camera position in real time.
In an embodiment of the present invention, in step S1, for each descriptor of the reference photo, the most similar one descriptor is found in the photos to be matched; extracting N descriptors from the reference picture to obtain the matching of N pairs of feature points; and according to the difference sorting of the descriptors, keeping the set match with the minimum difference. The characteristic points are places with severe light and shade changes in the picture, FAST corner points are used as the characteristic points, descriptors represent visual characteristics around the characteristic points, and ORB descriptors are used.
In an embodiment of the present invention, in step S2, the process of calculating the position of the 3d point includes:
step S21, obtaining a transformation matrix of the picture to be matched relative to the reference picture by using a five-point method;
step S22, using a triangularization method to obtain the position of the 3d point with respect to the reference picture.
In an embodiment of the present invention, in step S3, the matching method includes:
step S31, descriptor matching: for each descriptor of the reference photo, finding the most similar one in the photos to be matched; extracting N descriptors from the reference picture to obtain the matching of N pairs of feature points; according to the difference ordering of the descriptors, keeping M matching pairs with the minimum difference;
step S32, using RANSAC algorithm to screen out the wrong pairings among these pairings, resulting in the final pairing.
In an embodiment of the present invention, in step S5, the position t _1_3 and the orientation R _1_3 of the third picture relative to the first picture are calculated using a visual odometer. The transformation matrix of the local map relative to the global map is marked as T _ w _ 1; in step S6, the transformation matrix T _ w _1 is used to obtain the position T _ w _3 of the third picture relative to the map, i.e., T _ w _1_ T _1_ 3.
In an embodiment of the present invention, in step S7, after receiving the subsequent photo, the position t _1_ n of the current camera relative to the reference photo is obtained by the visual odometer; and obtaining the position of the current camera relative to the map through T _ w _ n-T _ w _ 1T _1_ n.
FIG. 5 is a flowchart of a visual map positioning method according to an embodiment of the present invention; referring to fig. 5, in an embodiment of the present invention, the method for positioning a visual map includes the following steps:
The feature point is a place with severe brightness change in the picture, the FAST corner point is used as the feature point, the descriptor represents the visual feature around the feature point, and the ORB descriptor is used here.
And 2, calculating the positions and the orientations of the feature points relative to the first picture. These 3d points of known location constitute a local map as illustrated in fig. 6. The process of calculating the position of the 3d point comprises the following steps:
(1) obtaining a transformation matrix of t2 relative to t1 by using a five-point method; and inputting to obtain feature point pairs in the last step.
(2) The position of the 3d point relative to t1 is obtained by using a triangulation method; the input is the transformation matrix of t2 versus t1 obtained in the previous step.
And 3, finding the matching relation between the local map and the global map (positioning map) through the descriptor matching from 3d to 3 d.
The matching method comprises the following steps:
(1) descriptor matching: the method matches the descriptors between t1 and t2 before; resulting in M matched pairs of 3d points.
(2) The RANSAC is used to screen out the wrong pairings in these pairings, resulting in the final pairing.
And 4, calculating a transformation matrix T _ w _1 from the local map point to the global map, as shown in FIG. 7. The input is the pairing between the resulting local and global map 3d points of the previous step, as shown in fig. 8.
And 5, calculating the position (t _1_3) and the orientation (R _1_3) of the third picture relative to the first picture by using the visual odometer.
And 6, obtaining the position of the third picture relative to the map by using the transformation matrix T _ w _ 1: t _ w _3 — T _ w _1_ T _1_ 3.
And 7, after receiving the subsequent pictures, immediately obtaining the position of the current camera relative to the first picture through a visual odometer: t _1_ n; and obtaining the position of the current camera relative to the map through T _ w _ n-T _ w _ 1T _1_ n.
And 8, simultaneously establishing more 3d points of the local map by the subsequent picture and the previous picture. These 3d points can be matched with the global map to obtain more accurate T _ w _ 1.
FIG. 9 is a schematic diagram of a visual map positioning system according to an embodiment of the present invention; referring to fig. 9, in an embodiment of the present invention, the visual map positioning system includes: the device comprises a transformation matrix acquisition module 1, a relative position and orientation acquisition module 3 and a data transformation module 5. The transformation matrix acquisition module 1 is used for calculating a transformation matrix of the local map relative to the global map through descriptor matching of the local map 3d point and the global map 3d point; the relative position and orientation acquisition module 3 is used for acquiring the position and orientation of the set picture relative to the reference picture; the data transformation module 5 is used to transform the visual odometry derived position and orientation into a global map using the transformation matrix.
FIG. 10 is a block diagram of a visual map positioning system according to an embodiment of the present invention; referring to fig. 10, in an embodiment of the present invention, the transformation matrix obtaining module 1 includes: a feature and description extracting unit 11, a position and orientation acquiring unit 13, a matching relationship acquiring unit 15, and a transformation matrix calculating unit 17. The feature and description extracting unit 11 is configured to obtain a reference photo and a photo to be matched, and extract feature points and descriptors of the reference photo and the photo to be matched. The position and orientation obtaining unit 13 is used for calculating the position and orientation of the feature points of the picture to be matched relative to the feature points of the reference picture; the 3d points of known location constitute a local map. The matching relationship obtaining unit 15 is configured to find a matching relationship between the local map and the global map by descriptor matching from the local map 3d point to the global map 3d point. The transformation matrix calculation unit 17 is used to calculate a transformation matrix from the local map points to the global map.
In an embodiment of the present invention, the calculation manner of the transformation matrix calculation unit is as follows: setting a global transformation to be composed of a rotation amount R and a translation amount t (in an embodiment of the present invention, the global transformation includes others); the following linear equations are listed: p2 ═ R × p1+ t; listing a corresponding number of linear equations according to the logarithm of (p1, p 2); due to the many pairs (p1, p2), the unknowns R and t are obtained by solving a solution to the system of linear equations.
Referring to fig. 10, in an embodiment of the present invention, the data transformation module 5 includes: a relative map position acquisition unit 51 and a camera relative position acquisition unit 53. The relative map position obtaining unit 51 is used for obtaining the position of the corresponding picture relative map by using the transformation matrix; the camera relative position obtaining unit 53 is configured to obtain, through the visual odometer, a position of the current camera relative to the reference picture after receiving the subsequent picture, and then obtain a position of the current camera relative to the map.
In an embodiment of the present invention, for each descriptor of a reference photo, the feature and description extraction unit 11 finds the most similar one descriptor among the photos to be matched; extracting N descriptors from the reference picture to obtain the matching of N pairs of feature points; and according to the difference sorting of the descriptors, keeping the set match with the minimum difference. The position and orientation obtaining unit 13 obtains a transformation matrix of the picture to be matched relative to the reference picture by using a five-point method, and obtains the position of the 3d point relative to the reference picture by using a triangulation method.
In an embodiment of the present invention, the matching relation obtaining unit 15 includes: descriptor matching subunit, pairing subunit. The descriptor matching subunit is used for finding the most similar descriptor in the photos to be matched for each descriptor of the reference photos; extracting N descriptors from the reference picture to obtain the matching of N pairs of feature points; and according to the difference sorting of the descriptors, keeping M matching pairs with the minimum difference. The pairing sub-unit screens out the wrong pairing in the pairings by using RANSAC algorithm to obtain the final pairing.
In an embodiment of the present invention, the relative position and orientation acquisition module 3 comprises a visual odometer; the visual odometer calculates the position t _1_3 and orientation R _1_3 of the third picture relative to the first picture; the transformation matrix of the local map relative to the global map is denoted as T _ w _ 1. The relative map position obtaining unit 51 obtains the position T _ w _3 of the third picture relative map by using the transformation matrix T _ w _1 as T _1_ 3. The camera relative position obtaining unit 53 obtains the position T _ w _ n of the current camera relative map, which is T _ w _1 × T _1_ n.
In summary, the visual map positioning method and system provided by the invention can obtain the estimated value of the current camera position with very low delay, and update the camera position to obtain a more accurate position after the matching of 3d to 2d with longer operation time is finished, thereby reducing the delay of positioning.
In the invention, the visual odometer does not depend on the result of map matching, so the positions of all pictures can be quickly obtained through the visual odometer. And after the parallel map matching is finished, updating the position of the latest picture by using the map matching result. Compared with a Kalman filtering-based method, the method has better parallelism. The invention also provides possibility for a map matching method with better use performance but lower speed in the future.
The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The description and applications of the invention herein are illustrative and are not intended to limit the scope of the invention to the embodiments described above. Variations and modifications of the embodiments disclosed herein are possible, and alternative and equivalent various components of the embodiments will be apparent to those skilled in the art. It will be clear to those skilled in the art that the present invention may be embodied in other forms, structures, arrangements, proportions, and with other components, materials, and parts, without departing from the spirit or essential characteristics thereof. Other variations and modifications of the embodiments disclosed herein may be made without departing from the scope and spirit of the invention.
Claims (21)
1. A visual map positioning method, characterized in that the visual map positioning method comprises:
calculating a transformation matrix of the local map relative to the global map by matching descriptors of the local map 3d point and the global map 3d point;
acquiring the position and orientation of the set photo relative to the reference photo;
and transforming the position and orientation of the acquired setting photo relative to the reference photo into a global map by using the transformation matrix.
2. The visual map positioning method of claim 1, wherein:
the process of calculating the transformation matrix of the local map relative to the global map comprises:
s1, acquiring a reference photo and a photo to be matched, and extracting feature points and descriptors of the reference photo and the photo to be matched;
step S2, calculating the position and orientation of the feature point of the picture to be matched relative to the feature point of the reference picture; the 3d points with known positions form a local map;
step S3, finding out the matching relationship between the local map and the global map through the descriptor matching from the local map 3d point p1 to the global map 3d point p 2;
and step S4, calculating a transformation matrix from the local map points to the global map.
3. The visual map positioning method of claim 2, wherein:
in step S4, the global transformation is made up of a rotation amount R and a translation amount t;
listing the corresponding number of linear equations p 2R p1+ t according to the logarithm of (p1, p 2); the unknowns R and t are obtained by solving the solution of the system of linear equations.
4. The visual map positioning method of claim 1, wherein:
the process of acquiring the position and orientation of the setting photograph with respect to the reference photograph includes step S5: the position and orientation of the corresponding picture relative to the reference photograph are calculated using a visual odometer.
5. The visual map positioning method of claim 1, wherein:
the process of transforming the position and orientation of the acquired setting photo relative to the reference photo into the global map includes:
step S6, obtaining the position of the corresponding picture relative to the map by using the transformation matrix;
and step S7, after receiving the subsequent pictures, obtaining the position of the current camera relative to the reference picture through the visual odometer, and then obtaining the position of the current camera relative to the map.
6. The visual map positioning method of claim 4, wherein:
the process of transforming the position and orientation of the acquired setting photo relative to the reference photo into the global map further includes step S8, the subsequent picture and the existing picture together establishing more 3d points of the local map; and after the 3d point of the local map is matched with the global map, a more accurate transformation matrix can be obtained.
7. The visual map positioning method of claim 2, wherein:
in step S1, for each descriptor of the reference photo, the most similar one is found among the photos to be matched; extracting N descriptors from the reference picture to obtain the matching of N pairs of feature points; and according to the difference sorting of the descriptors, keeping the set match with the minimum difference.
8. A visual map localization method according to claim 2 or 6, characterized in that:
in step S1, the feature points are places with severe light and shade changes in the picture, FAST corner points are used as the feature points, descriptors represent visual features around the feature points, and ORB descriptors are used.
9. The visual map positioning method of claim 2, wherein:
in step S2, the process of calculating the position of the 3d point includes:
s21, obtaining a transformation matrix of the picture to be matched relative to the reference picture by using a five-point method;
step S22, the position of the 3d point relative to the reference picture is obtained using the triangulation method.
10. The visual map positioning method of claim 2, wherein:
in step S3, the matching method includes:
step S31, descriptor matching: for each descriptor of the reference photo, finding the most similar one in the photos to be matched; extracting N descriptors from the reference picture to obtain the matching of N pairs of feature points; according to the difference ordering of the descriptors, keeping M matching pairs with the minimum difference;
and step S32, screening out wrong pairings in the pairings by using a RANSAC algorithm to obtain a final pairing.
11. The visual map positioning method of claim 4, wherein:
in step S5, calculating a position t _1_3 and an orientation R _1_3 of the third picture relative to the first picture using a visual odometer;
the transformation matrix of the local map relative to the global map is marked as T _ w _ 1; in step S6, the transformation matrix T _ w _1 is used to obtain the position T _ w _3 of the third picture relative to the map, i.e., T _ w _1_ T _1_ 3.
12. The visual map positioning method of claim 4, wherein:
in step S7, after receiving the subsequent photo, the position t _1_ n of the current camera relative to the reference photo is obtained by the visual odometer; and obtaining the position of the current camera relative to the map through T _ w _ n-T _ w _ 1T _1_ n.
13. A visual map positioning system, comprising:
the transformation matrix acquisition module is used for calculating a transformation matrix of the local map relative to the global map through descriptor matching of the local map 3d point and the global map 3d point;
the relative position and orientation acquisition module is used for acquiring the position and orientation of the set photo relative to the reference photo;
and the data transformation module is used for transforming the position and the orientation obtained by the visual odometer into a global map by using the transformation matrix.
14. A visual map positioning system as claimed in claim 13, wherein:
the transformation matrix obtaining module includes:
a feature and description extraction unit for acquiring the reference picture and the picture to be matched, and extracting feature points and descriptors of the reference picture and the picture to be matched;
a position and orientation acquisition unit for calculating the position and orientation of the feature points of the picture to be matched with respect to the feature points of the reference picture; the 3d points with known positions form a local map;
a matching relationship obtaining unit, configured to find a matching relationship between the local map and the global map through descriptor matching from the local map 3d point to the global map 3d point;
-a transformation matrix calculation unit to calculate a transformation matrix from the local map points to the global map.
15. A visual map positioning system as claimed in claim 14, wherein:
the calculation mode of the transformation matrix calculation unit is as follows:
the global transformation consists of a rotation amount R and a translation amount t;
listing the corresponding number of linear equations p 2R p1+ t according to the logarithm of (p1, p 2); the unknowns R and t are obtained by solving the solution of the system of linear equations.
16. A visual map positioning system as claimed in claim 13, wherein:
the relative position and orientation acquisition module comprises a visual odometer; the data transformation module includes:
-a relative map position obtaining unit to obtain a position of the corresponding picture relative to the map using the transformation matrix;
a camera relative position acquisition unit for obtaining, by means of a visual odometer, the position of the current camera relative to the reference photograph and then to the map, after receiving the subsequent pictures.
17. A visual map positioning system as claimed in claim 14, wherein:
for each descriptor of a reference photo, the feature and description extraction unit finds the most similar descriptor in the photos to be matched; extracting N descriptors from the reference picture to obtain the matching of N pairs of feature points; and according to the difference sorting of the descriptors, keeping the set match with the minimum difference.
18. A visual map positioning system as claimed in claim 14, wherein:
the position and orientation acquisition unit obtains a transformation matrix of the picture to be matched relative to the reference picture by using a five-point method, and obtains the position of the 3d point relative to the reference picture by using a triangulation method.
19. A visual map positioning system as claimed in claim 14, wherein:
the matching relationship acquisition unit includes:
-a descriptor matching subunit to find, for each descriptor of a reference photograph, the most similar one of the descriptors in the photographs to be matched; extracting N descriptors from the reference picture to obtain the matching of N pairs of feature points; according to the difference ordering of the descriptors, keeping M matching pairs with the minimum difference;
a pairing subunit, which screens out the wrong pairings among these pairings using the RANSAC algorithm, resulting in the final pairing.
20. A visual map positioning system as claimed in claim 16, wherein:
the relative position and orientation acquisition module calculates the position t _1_3 and the orientation R _1_3 of the third picture relative to the first picture; the transformation matrix of the local map relative to the global map is marked as T _ w _ 1;
the relative map position obtaining unit obtains a position T _ w _3 of the third picture relative map by using the transformation matrix T _ w _1, namely T _ w _1, T _1_ 3.
21. A visual map positioning system as claimed in claim 16, wherein:
the camera relative position acquisition unit acquires a position T _ w _ n of the current camera relative map, which is T _ w _1 × T _1_ n.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910877005.5A CN110647609B (en) | 2019-09-17 | 2019-09-17 | Visual map positioning method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910877005.5A CN110647609B (en) | 2019-09-17 | 2019-09-17 | Visual map positioning method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110647609A true CN110647609A (en) | 2020-01-03 |
| CN110647609B CN110647609B (en) | 2023-07-18 |
Family
ID=69010627
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910877005.5A Active CN110647609B (en) | 2019-09-17 | 2019-09-17 | Visual map positioning method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110647609B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113029128A (en) * | 2021-03-25 | 2021-06-25 | 浙江商汤科技开发有限公司 | Visual navigation method and related device, mobile terminal and storage medium |
| WO2022252237A1 (en) * | 2021-06-04 | 2022-12-08 | 华为技术有限公司 | Methods and apparatus for encoding and decoding 3d map |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130073389A1 (en) * | 2011-09-15 | 2013-03-21 | Stephan HEATH | System and method for providing sports and sporting events related social/geo/promo link promotional data sets for end user display of interactive ad links, promotions and sale of products, goods, gambling and/or services integrated with 3d spatial geomapping, company and local information for selected worldwide locations and social networking |
| CN106679648A (en) * | 2016-12-08 | 2017-05-17 | 东南大学 | Vision-inertia integrated SLAM (Simultaneous Localization and Mapping) method based on genetic algorithm |
| CN107084727A (en) * | 2017-04-12 | 2017-08-22 | 武汉理工大学 | A kind of vision positioning system and method based on high-precision three-dimensional map |
| CN107784671A (en) * | 2017-12-01 | 2018-03-09 | 驭势科技（北京）有限公司 | A kind of method and system positioned immediately for vision with building figure |
| CN108072370A (en) * | 2016-11-18 | 2018-05-25 | 中国科学院电子学研究所 | Robot navigation method based on global map and the robot with this method navigation |
| CN108682027A (en) * | 2018-05-11 | 2018-10-19 | 北京华捷艾米科技有限公司 | VSLAM realization method and systems based on point, line Fusion Features |
| CN108717712A (en) * | 2018-05-29 | 2018-10-30 | 东北大学 | A kind of vision inertial navigation SLAM methods assumed based on ground level |
| US20180328753A1 (en) * | 2017-05-09 | 2018-11-15 | Raven Telemetry Inc. | Local location mapping method and system |
| CN109211251A (en) * | 2018-09-21 | 2019-01-15 | 北京理工大学 | A kind of instant positioning and map constructing method based on laser and two dimensional code fusion |
| CN109255749A (en) * | 2017-07-12 | 2019-01-22 | 来福机器人 | From the map structuring optimization in non-autonomous platform of advocating peace |
| CN109307508A (en) * | 2018-08-29 | 2019-02-05 | 中国科学院合肥物质科学研究院 | A kind of panorama inertial navigation SLAM method based on more key frames |
| US20190065863A1 (en) * | 2017-08-23 | 2019-02-28 | TuSimple | Feature matching and correspondence refinement and 3d submap position refinement system and method for centimeter precision localization using camera-based submap and lidar-based global map |
| CN109615645A (en) * | 2018-12-07 | 2019-04-12 | 国网四川省电力公司电力科学研究院 | The Feature Points Extraction of view-based access control model |
| CN110017841A (en) * | 2019-05-13 | 2019-07-16 | 大有智能科技(嘉兴)有限公司 | Vision positioning method and its air navigation aid |
| CN110118554A (en) * | 2019-05-16 | 2019-08-13 | 深圳前海达闼云端智能科技有限公司 | SLAM method, apparatus, storage medium and device based on visual inertia |
-
2019
- 2019-09-17 CN CN201910877005.5A patent/CN110647609B/en active Active
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130073389A1 (en) * | 2011-09-15 | 2013-03-21 | Stephan HEATH | System and method for providing sports and sporting events related social/geo/promo link promotional data sets for end user display of interactive ad links, promotions and sale of products, goods, gambling and/or services integrated with 3d spatial geomapping, company and local information for selected worldwide locations and social networking |
| CN108072370A (en) * | 2016-11-18 | 2018-05-25 | 中国科学院电子学研究所 | Robot navigation method based on global map and the robot with this method navigation |
| CN106679648A (en) * | 2016-12-08 | 2017-05-17 | 东南大学 | Vision-inertia integrated SLAM (Simultaneous Localization and Mapping) method based on genetic algorithm |
| CN107084727A (en) * | 2017-04-12 | 2017-08-22 | 武汉理工大学 | A kind of vision positioning system and method based on high-precision three-dimensional map |
| US20180328753A1 (en) * | 2017-05-09 | 2018-11-15 | Raven Telemetry Inc. | Local location mapping method and system |
| CN109255749A (en) * | 2017-07-12 | 2019-01-22 | 来福机器人 | From the map structuring optimization in non-autonomous platform of advocating peace |
| US20190065863A1 (en) * | 2017-08-23 | 2019-02-28 | TuSimple | Feature matching and correspondence refinement and 3d submap position refinement system and method for centimeter precision localization using camera-based submap and lidar-based global map |
| CN107784671A (en) * | 2017-12-01 | 2018-03-09 | 驭势科技（北京）有限公司 | A kind of method and system positioned immediately for vision with building figure |
| CN108682027A (en) * | 2018-05-11 | 2018-10-19 | 北京华捷艾米科技有限公司 | VSLAM realization method and systems based on point, line Fusion Features |
| CN108717712A (en) * | 2018-05-29 | 2018-10-30 | 东北大学 | A kind of vision inertial navigation SLAM methods assumed based on ground level |
| CN109307508A (en) * | 2018-08-29 | 2019-02-05 | 中国科学院合肥物质科学研究院 | A kind of panorama inertial navigation SLAM method based on more key frames |
| CN109211251A (en) * | 2018-09-21 | 2019-01-15 | 北京理工大学 | A kind of instant positioning and map constructing method based on laser and two dimensional code fusion |
| CN109615645A (en) * | 2018-12-07 | 2019-04-12 | 国网四川省电力公司电力科学研究院 | The Feature Points Extraction of view-based access control model |
| CN110017841A (en) * | 2019-05-13 | 2019-07-16 | 大有智能科技(嘉兴)有限公司 | Vision positioning method and its air navigation aid |
| CN110118554A (en) * | 2019-05-16 | 2019-08-13 | 深圳前海达闼云端智能科技有限公司 | SLAM method, apparatus, storage medium and device based on visual inertia |
Non-Patent Citations (2)
| Title |
|---|
| ADRIAN GARCEA 等: ""Robust Map Alignment for Cooperative Visual SLAM"", 《2018 25TH IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP)》 * |
| 潘林豪 等: ""基于直接法的视觉同时定位与地图构建技术综述"", 《计算机应用研究》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113029128A (en) * | 2021-03-25 | 2021-06-25 | 浙江商汤科技开发有限公司 | Visual navigation method and related device, mobile terminal and storage medium |
| CN113029128B (en) * | 2021-03-25 | 2023-08-25 | 浙江商汤科技开发有限公司 | Visual navigation method and related device, mobile terminal and storage medium |
| WO2022252237A1 (en) * | 2021-06-04 | 2022-12-08 | 华为技术有限公司 | Methods and apparatus for encoding and decoding 3d map |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110647609B (en) | 2023-07-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9898856B2 (en) | Systems and methods for depth-assisted perspective distortion correction | |
| US10410397B2 (en) | Picture presentation method and apparatus | |
| KR101121034B1 (en) | System and method for obtaining camera parameters from multiple images and computer program products thereof | |
| WO2020014909A1 (en) | Photographing method and device and unmanned aerial vehicle | |
| CN107316326B (en) | Edge-based disparity map calculation method and device applied to binocular stereo vision | |
| WO2020007320A1 (en) | Method for fusing multi-visual angle images, apparatus, computer device, and storage medium | |
| CN107690673B (en) | Image processing method and device and server | |
| CN110069593B (en) | Image processing method and system, server, computer readable medium | |
| CN107146200B (en) | Unmanned aerial vehicle remote sensing image splicing method based on image splicing quality evaluation | |
| CN104685513A (en) | Feature based high resolution motion estimation from low resolution images captured using an array source | |
| CN111127524A (en) | Method, system and device for tracking trajectory and reconstructing three-dimensional image | |
| WO2020083307A1 (en) | Method, apparatus, and storage medium for obtaining depth image | |
| CN110647609B (en) | Visual map positioning method and system | |
| CN108401460B (en) | Method, system, storage medium, and computer program product for generating a disparity map | |
| CN115329111A (en) | Image feature library construction method and system based on point cloud and image matching | |
| Wang et al. | Robust color correction in stereo vision | |
| CN112270748B (en) | Three-dimensional reconstruction method and device based on image | |
| US9811889B2 (en) | Method, apparatus and computer program product for generating unobstructed object views | |
| CN110796706A (en) | Visual positioning method and system | |
| CN113129422A (en) | Three-dimensional model construction method and device, storage medium and computer equipment | |
| CN113298871B (en) | Map generation method, positioning method, system thereof, and computer-readable storage medium | |
| CN106548482B (en) | Dense matching method and system based on sparse matching and image edges | |
| CN114387532A (en) | Boundary identification method and device, terminal, electronic equipment and unmanned equipment | |
| US10965866B2 (en) | Image generation system, image display system, image generation method, and moving vehicle | |
| KR100953737B1 (en) | System for drawing manhole using image matching |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20201104 Address after: Unit 908, 9 / F, building 1, No. 58, Xiangke Road, China (Shanghai) pilot Free Trade Zone, Pudong New Area, Shanghai, 200123 Applicant after: Shanghai Zhizhuo Information Technology Co.,Ltd. Address before: 200120 9 / F, building a, No. 58 Xiangke Road, Pudong New Area, Shanghai Applicant before: SHANGHAI TUQU INFORMATION TECHNOLOGY Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20210407 Address after: Unit 908, 9 / F, building 1, 58 Xiangke Road, China (Shanghai) pilot Free Trade Zone, Pudong New Area, Shanghai, 200120 Applicant after: SHANGHAI TUQU INFORMATION TECHNOLOGY Co.,Ltd. Address before: 200123 unit 908, 9 / F, building 1, 58 Xiangke Road, China (Shanghai) pilot Free Trade Zone, Pudong New Area, Shanghai Applicant before: Shanghai Zhizhuo Information Technology Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |