CN112581514A - Map construction method and device and storage medium - Google Patents
Map construction method and device and storage medium Download PDFInfo
- Publication number
- CN112581514A CN112581514A CN201910944288.0A CN201910944288A CN112581514A CN 112581514 A CN112581514 A CN 112581514A CN 201910944288 A CN201910944288 A CN 201910944288A CN 112581514 A CN112581514 A CN 112581514A
- Authority
- CN
- China
- Prior art keywords
- map
- image frame
- sub
- current
- current image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000010276 construction Methods 0.000 title claims abstract description 22
- 230000000007 visual effect Effects 0.000 claims abstract description 70
- 238000000034 method Methods 0.000 claims abstract description 40
- 230000004044 response Effects 0.000 claims abstract description 35
- 238000013507 mapping Methods 0.000 claims description 32
- 238000004590 computer program Methods 0.000 claims description 8
- 238000012423 maintenance Methods 0.000 claims 1
- 238000010586 diagram Methods 0.000 description 23
- 238000012545 processing Methods 0.000 description 4
- 238000005457 optimization Methods 0.000 description 3
- 230000033001 locomotion Effects 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V10/757—Matching configurations of points or features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Abstract
The present disclosure provides a map construction method, a map construction device, and a storage medium, wherein the map construction method includes: performing visual tracking on the current image frame in combination with the current sub-map to obtain a visual tracking result; determining whether the current image frame and the current sub map have a common-view relationship according to a visual tracking result; and in response to the current image frame and the current sub-map having no co-view relationship, creating a new sub-map. According to the method and the device, when the current image frame and the current sub-map do not have a common-view relation, a new sub-map is created to solve the problem that once the vision in a single map is difficult to continue tracking, the map cannot be expanded. Meanwhile, the method provided by the disclosure fully utilizes visual information, does not depend on the assistance of IMU and other sensor information, and can be applied to wider fields.
Description
Technical Field
The present disclosure relates to the field of map processing, and in particular, to a map construction method and apparatus, and a storage medium.
Background
In a SLAM (Simultaneous Localization And Mapping) system, map construction is an important component. The map records scene information accessed once in some form. Consistent maps may reduce errors in the map construction process.
Maintaining consistent maps is a difficult task, however, especially on mobile devices such as cell phones. Due to poor quality of images acquired by a camera, image motion blur caused by over-fast movement of a mobile phone video by a user, low precision of an Inertial Measurement Unit (IMU) installed on mobile phone equipment and the like, great deviation of poses of partial key frames of the acquired images can be caused, and therefore a map cannot be expanded normally.
Disclosure of Invention
In view of the above, the present disclosure provides a map construction method and apparatus, and a storage medium, to solve the deficiencies in the related art.
According to a first aspect of the embodiments of the present disclosure, there is provided a map construction method, the method including: performing visual tracking on the current image frame in combination with the current sub-map to obtain a visual tracking result; determining whether the current image frame and the current sub map have a common-view relationship according to the visual tracking result; and in response to the current image frame and the current sub-map having no co-view relationship, creating a new sub-map.
According to the embodiment, when the current image frame and the current sub-map do not have the common-view relationship, a new sub-map is created, and the problem that once the vision in a single map is difficult to continue tracking, the map cannot be expanded can be solved.
Optionally, the determining whether the current image frame and the current sub-map have a common view relationship according to the visual tracking result includes: and in response to the visual tracking result indicating that matched feature points exist between the current image frame and the current sub-map and the number of matched feature points is greater than or equal to a first preset threshold value, determining that a common-view relationship exists between the current image frame and the current sub-map, otherwise, determining that the common-view relationship does not exist between the current image frame and the current sub-map.
In the embodiment, the common-view relationship between the current image frame and the current sub-map is determined according to the fact that the matched feature point between the current image frame and the current sub-map is larger than or equal to the first preset threshold, otherwise, the current image frame and the current sub-map are determined not to have the common-view relationship, and the method is simple and convenient to implement and high in usability.
Optionally, the determining whether the current image frame and the current sub-map have a common view relationship according to the visual tracking result includes: in response to the visual tracking result indicating that there are matched feature points between the current image frame and the current sub-map, removing a first feature point which is in a wrong match from the matched feature points, and obtaining a second feature point which is in a valid match; and in response to the number of the second characteristic points being greater than or equal to a second preset threshold, determining that the current image frame and the current sub-map have a common-view relationship, otherwise determining that the current image frame and the current sub-map do not have a common-view relationship.
In the embodiment, in addition to determining whether the matched feature points exist between the current image frame and the current sub-map, whether the common-view relationship exists between the current image frame and the current sub-map is determined according to the number of the second feature points which are effectively matched between the current image frame and the current sub-map, so that the possibility of misjudgment is reduced.
Optionally, the obtaining a second feature point that is effectively matched after removing the first feature point that is incorrectly matched from the matched feature points includes: inputting the current image frame and the image frame included in the current sub-map into a pre-established feature point matching model, and removing the first feature points which are in error matching from the matched feature points through the feature point matching model to obtain the second feature points which are in effective matching.
In the above embodiment, the feature point matching model may be pre-established, the current image frame and the current sub-map are input into the feature point matching model, and the feature point matching model removes the first feature point of the error matching between the current image frame and the current sub-map, so as to obtain the second feature point of the effective matching, thereby quickly determining the second feature point of the effective matching, and achieving the convenience.
Optionally, the method further comprises: and in response to the common-view relationship between the current image frame and the current sub-map, expanding the current sub-map by using the current image frame.
In the above embodiment, when the current image frame and the current sub-map have a common view relationship, the current sub-map may be expanded based on the current image frame, so as to improve the map accuracy.
Optionally, after the expanding the current sub-map with the current image frame, the method further includes: in response to a co-view relationship being provided between the current image frame and at least one of the other sub-maps, merging the current sub-map with at least one of the other sub-maps.
In the above embodiment, when the current image frame and at least one other sub-map have a common view relationship, the current sub-map and the at least one other sub-map may be merged to improve the map accuracy.
Optionally, the creating a new sub-map includes: creating a new sub-map with empty content; and in the new sub-map with empty content, taking the current image frame as an initial frame of the new sub-map.
In the embodiment, the current image frame can be used as the initial frame of the new sub-map in the new sub-map with empty content, so that the new sub-map can be expanded according to the collected new current image frame in the following process, and the problem that the map cannot be expanded once a single map is adopted in the related art and the vision is difficult to track can be avoided.
Optionally, after creating the new sub-map, the method further includes: carrying out visual tracking on the new current image frame in combination with the new sub-map to obtain a new visual tracking result; determining whether the new current image frame and the new sub-map have a common-view relationship according to the new visual tracking result; and in response to the new current image frame and the new sub-map having no co-view relationship, creating a new sub-map again.
In the above embodiment, the new current image frame may be combined with the new sub-map for visual tracking, and according to the new visual tracking result, it is determined whether the new current image frame and the new sub-map have a common view relationship, and if the new current image frame and the new sub-map do not have the common view relationship, the new sub-map may still be created again, thereby avoiding a problem that the map cannot be expanded once the vision is difficult to track due to the adoption of a single map in the related art.
Optionally, the method further comprises: and merging all sub-maps meeting preset merging conditions to obtain the global map.
In the above embodiment, all the sub-maps meeting the preset merging condition may be merged to obtain the global map, so that the usability is higher.
Optionally, the sub-map satisfying the preset merging condition is determined in the following manner: in response to the common-view relationship between at least two sub-maps, determining that the at least two sub-maps with the common-view relationship are sub-maps meeting a preset merging condition.
In the above embodiment, when the at least two sub-maps have a common view relationship, it may be determined that the at least two sub-maps satisfy a preset merging condition, so as to ensure accuracy of the finally obtained global map.
According to a second aspect of embodiments of the present disclosure, there is provided a multi-map construction apparatus, the apparatus including: the first visual tracking module is used for carrying out visual tracking on the current image frame in combination with the current sub-map to obtain a visual tracking result; the first co-view relationship determining module is used for determining whether the current image frame and the current sub-map have a co-view relationship according to the visual tracking result; and the first map building module is used for responding to that the current image frame and the current sub map do not have a common view relation, and then creating a new sub map.
Optionally, the first common view relationship determining module includes: a first determining sub-module, configured to determine that there is a common-view relationship between the current image frame and the current sub-map if the visual tracking result indicates that there are matched feature points between the current image frame and the current sub-map and the number of matched feature points is greater than or equal to a first preset threshold, and otherwise determine that there is no common-view relationship between the current image frame and the current sub-map.
Optionally, the first common view relationship determining module includes: the obtaining sub-module is used for responding to the visual tracking result to indicate that matched feature points exist between the current image frame and the current sub-map, removing first feature points which are in error matching from the matched feature points, and obtaining second feature points which are in effective matching; and the second determining sub-module is used for determining that the current image frame and the current sub-map have a common-view relationship in response to the number of the second characteristic points being greater than or equal to a second preset threshold, and otherwise determining that the current image frame and the current sub-map do not have the common-view relationship.
Optionally, the obtaining sub-module includes: and the obtaining unit is used for inputting the current image frame and the image frame included in the current sub-map into a pre-established feature point matching model, and removing the first feature points which are in error matching from the matched feature points through the feature point matching model to obtain the second feature points which are effectively matched.
Optionally, the apparatus further comprises: and the second map building module is used for responding to the common view relation between the current image frame and the current sub map, and expanding the current sub map by using the current image frame.
Optionally, the apparatus further comprises: and the third map building module is used for combining the current sub-map and at least one other sub-map in response to the common view relation between the current image frame and the other sub-map.
Optionally, the first mapping module comprises: the first map building sub-module is used for creating a new sub-map with empty content; and the second map construction sub-module is used for taking the current image frame as an initial frame of the new sub-map in the new sub-map with empty content.
Optionally, the apparatus further comprises: the second visual tracking module is used for carrying out visual tracking on the new current image frame in combination with the new sub-map to obtain a new visual tracking result; a second common-view relationship determining module, configured to determine whether a common-view relationship exists between the new current image frame and the new sub-map according to the new visual tracking result; and the fourth map building module is used for creating a new sub map again in response to the new current image frame and the new sub map not having the common view relation.
Optionally, the apparatus further comprises: and the fifth map building module is used for merging all sub-maps meeting the preset merging conditions to obtain the global map.
Optionally, the fifth mapping module includes: and the third determining sub-module is used for responding to the common-view relationship between the at least two sub-maps, and then determining that the at least two sub-maps with the common-view relationship are the sub-maps meeting the preset merging condition.
According to a third aspect of embodiments of the present disclosure, there is provided a computer-readable storage medium storing a computer program for executing the map construction method according to the first aspect described above.
According to a fourth aspect of embodiments of the present disclosure, there is provided a map building apparatus, the apparatus including: a processor; a memory for storing processor-executable instructions; the processor is configured to call the executable instructions stored in the memory to implement the map building method according to the first aspect.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.
FIG. 1 is a flowchart of a map building method shown in accordance with an exemplary embodiment of the present disclosure;
FIG. 2 is a flow chart of another mapping method illustrated by the present disclosure in accordance with an exemplary embodiment;
FIG. 3 is a flow chart of another mapping method shown in the present disclosure in accordance with an exemplary embodiment;
FIG. 4 is a flow chart of another mapping method shown in the present disclosure in accordance with an exemplary embodiment;
FIG. 5 is a flow chart of another mapping method shown in the present disclosure in accordance with an exemplary embodiment;
FIG. 6 is a flow chart of another mapping method shown in the present disclosure in accordance with an exemplary embodiment;
FIG. 7 is a flow chart of another mapping method shown in the present disclosure in accordance with an exemplary embodiment;
FIG. 8 is a block diagram of a map building apparatus shown in accordance with an exemplary embodiment of the present disclosure;
FIG. 9 is a block diagram of another mapping apparatus shown in accordance with an exemplary embodiment of the present disclosure;
FIG. 10 is a block diagram of another mapping apparatus shown in accordance with an exemplary embodiment of the present disclosure;
FIG. 11 is a block diagram of another mapping apparatus shown in accordance with an exemplary embodiment of the present disclosure;
FIG. 12 is a block diagram of another mapping apparatus shown in accordance with an exemplary embodiment of the present disclosure;
FIG. 13 is a block diagram of another mapping apparatus shown in accordance with an exemplary embodiment of the present disclosure;
FIG. 14 is a block diagram of another mapping apparatus shown in accordance with an exemplary embodiment of the present disclosure;
FIG. 15 is a block diagram of another mapping apparatus shown in accordance with an exemplary embodiment of the present disclosure;
FIG. 16 is a block diagram of another mapping apparatus shown in accordance with an exemplary embodiment of the present disclosure;
FIG. 17 is a block diagram of another mapping apparatus shown in accordance with an exemplary embodiment of the present disclosure;
FIG. 18 is a schematic structural diagram illustrating an apparatus for map construction according to an exemplary embodiment of the present disclosure.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.
The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as operated herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present disclosure. The word "if," as used herein, may be interpreted as "at … …" or "at … …" or "in response to a determination," depending on the context.
The embodiment of the present disclosure provides a map construction method, which may be used in a machine device, a terminal device, or a platform that needs to perform real-time map construction, and the following description only takes a terminal as an example. As shown in fig. 1, fig. 1 is a diagram construction method according to an exemplary embodiment, including the steps of:
in step 101, the current image frame is visually tracked in combination with the current sub-map, and a visual tracking result is obtained.
The terminal may acquire the current image frame through an image acquisition device, such as a camera. The current sub-map is a set of image information included in at least one previously acquired image frame.
Optionally, in this embodiment of the present disclosure, the terminal may further obtain corresponding sensor data through at least one other sensor, so that when the acquired image frame is subsequently visually tracked, the acquired image frame may be analyzed together with the sensor data, and accuracy of visually tracking the image frame is improved. In the embodiments of the present disclosure, the sensors may include, but are not limited to, an IMU sensor, an orientation sensor, an acceleration sensor, a gravity sensor, and the like.
Further, the terminal may perform visual tracking on the current image frame in combination with the current sub-map to obtain a visual tracking result.
In the embodiment of the present disclosure, for a current image frame and an image frame of a previous frame of the current image frame (i.e., a last image frame of a current sub-map is constructed), feature points are respectively extracted, it is determined whether a matched feature point exists between the current image frame and the previous image frame, if so, it is determined that a visual tracking result is the feature point which exists between the current image frame and the current sub-map, otherwise, it is determined that the visual tracking result is the feature point which does not exist between the current image frame and the current sub-map. In step 102, it is determined whether the current image frame and the current sub-map have a co-view relationship according to the visual tracking result.
The terminal may determine whether the current image frame and the current sub-map have a co-view relationship according to the visual tracking result indicating whether there is a matched feature point between the current image frame and the current sub-map.
Optionally, if the visual tracking result indicates that there are matched feature points between the current image frame and the current sub-map and the number of matched feature points is large, it is determined that there is a co-view relationship between the current image frame and the current sub-map, otherwise, it is determined that there is no co-view relationship between the current image frame and the current sub-map.
Or if the visual tracking result indicates that the current image frame and the current sub-map have matched feature points and the number of second feature points of effective matching (inlier) obtained after removing the first feature points of the error matching (outlier) from the matched feature points is more, determining that the current image frame and the current sub-map have a common-view relationship, otherwise determining that the current image frame and the current sub-map do not have the common-view relationship.
In step 103, in response to that there is no co-view relationship between the current image frame and the current sub-map, a new sub-map is created.
In response to that the current image frame and the current sub-map do not have a common view relationship, at this time, the terminal may create a new sub-map with empty content first, and then use the current image frame as an initial frame of the new sub-map, so as to continue to expand the new sub-map, thereby avoiding the problem that the map cannot be expanded once a single map is adopted in the related art and the vision is difficult to track.
In some alternative embodiments, the terminal may determine whether a co-view relationship exists between the current image frame and the current sub-map according to the visual tracking result in any one of the following manners.
The first mode > only judges whether the matched feature points exist between the current image frame and the current sub-map, and judges whether the number of the matched feature points between the current image frame and the current sub-map is larger than or equal to a first preset threshold value.
In the first mode, if matched feature points exist between the current image frame and the current sub-map, and the number of the matched feature points is greater than or equal to a first preset threshold value, the common-view relationship between the current image frame and the current sub-map is determined.
For example, the current sub-map constructs a scene A corresponding to the feature point corresponding to the last image frame a of the current sub-map, the corresponding feature point of the current image frame also corresponds to the scene A, the number of matched feature points of the current image frame and the image frame a is determined to be greater than or equal to a first preset threshold value, it is determined that enough matched feature points exist between the current image frame and the current sub-map, and the common view relationship between the current image frame and the current sub-map is determined.
For another example, a scene a corresponding to the last frame image frame a of the current sub-map is constructed, a scene B corresponding to the feature point corresponding to the current image frame is constructed, and the number of the determined matched feature points is smaller than a first preset threshold, which indicates that there are no sufficient number of matched feature points between the two.
In order to avoid the situation, in the embodiment of the present disclosure, the terminal may further determine whether the current image frame and the current sub-map have the common-view relationship by using a second method.
And the second mode simultaneously judges whether matched feature points exist between the current image frame and the current sub-map or not and judges whether the number of the feature points which are effectively matched between the current image frame and the current sub-map is larger than or equal to a second preset threshold value or not.
If there are matched feature points between the current image frame and the current sub-map, it is further required to further eliminate the first feature points of the error matching (outlier) through checking, such as geometric consistency, etc., to obtain the second feature points of the valid matching (inlier), and determine whether the number of the second feature points of the valid matching is sufficient, for example, whether it is greater than or equal to a second preset threshold. And when the number of the second characteristic points which are effectively matched is greater than or equal to a second preset threshold value, the terminal considers that the current image frame and the current sub-map have a common-view relationship. The second preset threshold and the first preset threshold have the same or different values, which is not limited in the present disclosure.
In other cases, for example, there is no matched feature point between the current image frame and the current sub-map, or although there is a matched feature point between the current image frame and the current sub-map, the number of second feature points that are effectively matched is small and smaller than the second preset threshold, it may also be considered that there is no co-view relationship between the current image frame and the current sub-map, so as to reduce the misjudgment.
In some optional embodiments, the current image frame and the current sub-map may be input into a pre-established feature point matching model, and the feature point matching model removes the first feature point with the error matching from the matched feature points, so as to obtain the second feature point with the effective matching.
The feature point matching model may be established based on a local geometric consistency or a global geometric consistency method, for example, the feature point matching model verifies all feature points between at least two frames of sample image frames by using the global geometric consistency method, a second feature point that is effectively matched between at least two frames of sample image frames has been marked, and the feature point matching model is trained so that the second feature point that is effectively matched between at least two sample image frames output by the feature point matching model and the second feature point that is marked in at least two frames of sample image frames satisfy or even agree with a preset fault tolerance range.
In the above embodiment, the feature point matching model may be pre-established, the current image frame and the current sub-map are input into the feature point matching model, and the feature point matching model removes the first feature point of the error matching between the current image frame and the current sub-map, so as to obtain the second feature point of the effective matching, thereby quickly determining the second feature point of the effective matching, and achieving the convenience.
In some alternative embodiments, such as shown in fig. 2, the method may further include the steps of:
in step 104, in response to the current image frame and the current sub-map having a common view relationship, the current sub-map is expanded by the current image frame.
In the embodiment of the present disclosure, if the current image frame and the current sub-map have a common view relationship, information of other feature points of the current image frame may be added to the current sub-map based on a feature point portion of the current image frame matched with the current sub-map, so as to expand the current sub-map.
In the above embodiment, when the current image frame and the current sub-map have a common view relationship, the current sub-map may be expanded based on the current image frame, so as to improve the map accuracy.
In some alternative embodiments, such as shown in fig. 3, after completing step 104, the method may further comprise:
in step 105, in response to the current image frame and the at least one other sub-map having a co-view relationship, the current sub-map is merged with the at least one other sub-map.
In this step, the terminal may also determine whether the current sub-map and at least one other sub-map except the current sub-map have a common-view relationship according to the manner of determining whether the current image frame and the current sub-map have the common-view relationship in step 102, so that when the current image frame and the at least one other sub-map have the common-view relationship, the current sub-map and the at least one other sub-map are merged by using a closed-loop algorithm, and an error is eliminated by global optimization, thereby improving the map accuracy.
In some alternative embodiments, such as shown in fig. 4, the step 103 may include:
in step 103-1, a new sub-map is created with empty content.
In the embodiment of the present disclosure, a new sub-map whose contents are empty may be created anew in addition to the current sub-map.
In step 103-2, in the new sub-map with empty content, the current image frame is used as the initial frame of the new sub-map.
And in the new sub-map with empty content, taking the current image frame as an initial frame of the new sub-map, wherein the new sub-map can comprise information of characteristic points in the current image frame.
In the above embodiment, the current image frame may be used as an initial frame of the new sub-map in the new sub-map with empty content, so that the new sub-map is subsequently expanded according to the collected new current image frame, thereby avoiding a problem that the map cannot be expanded once a single map is adopted in the related art and the vision is difficult to track.
In some alternative embodiments, such as shown in fig. 5, after completing step 103, the method may further comprise:
in step 106, the new current image frame is visually tracked in combination with the new sub-map, and a new visual tracking result is obtained.
And (4) performing visual tracking on the new current image frame acquired by the terminal by combining with the new sub-map according to the method in the step (101) again, so as to obtain a new visual tracking result.
In step 107, it is determined whether the new current image frame and the new sub-map have a co-view relationship according to the new visual tracking result.
In the embodiment of the present disclosure, it may be determined whether the new current image frame and the new sub-map have a common view relationship in the same manner as in step 102, which is not described herein again.
In step 108, in response to the new current image frame and the new sub-map having no co-view relationship, a new sub-map is created again.
If there is no co-view relationship between the new current image frame and the new sub-map, a new sub-map may be created again, and if there is a co-view relationship between the new current image frame and the new sub-map, the new sub-map may be extended by the new current image frame.
In the above embodiment, the new current image frame may be combined with the new sub-map for visual tracking, and according to the new visual tracking result, it is determined whether the new current image frame and the new sub-map have a common view relationship, and if the new current image frame and the new sub-map do not have the common view relationship, the new sub-map may still be created again, thereby avoiding a problem that the map cannot be expanded once the vision is difficult to track due to the adoption of a single map in the related art. In some alternative embodiments, such as shown in fig. 6, the method may further include:
in step 109, all the sub-maps meeting the preset merging condition are merged to obtain the global map.
In the disclosed embodiment, all sub-maps may be stored in the map database.
The terminal can determine whether the indication between the two sub-maps meets the preset merging condition by judging whether the common-view relationship exists between any two sub-maps, and can determine that the common-view relationship exists between the two sub-maps if the number of second characteristic points which are effectively matched between the two sub-maps is larger than or equal to a second preset threshold. Further, it may be determined that the two sub-maps satisfy a preset merging condition.
The terminal can correct the possible relative position offset between the two sub-maps by using the second feature points which are effectively matched between the two sub-maps with the common view relationship, and the two sub-maps are combined by adopting a closed loop algorithm to finally obtain the global map through global optimization.
In the above embodiment, when the at least two sub-maps have the common-view relationship, it may be determined that the at least two sub-maps satisfy the preset merging condition, and the sub-maps having the common-view relationship are merged by using the loop closing algorithm, so as to ensure the accuracy of the finally obtained global map and improve the usability.
In one embodiment, as shown in FIG. 7, FIG. 7 is another mapping method, shown according to an exemplary embodiment, including the steps of:
in step 201, the current image frame is visually tracked in combination with the current sub-map, and a visual tracking result is obtained.
In step 202, it is determined whether the current image frame and the current sub-map have a co-view relationship according to the visual tracking result.
In response to that the visual tracking result indicates that there are matched feature points between the current image frame and the current sub-map and the number of second feature points that are effectively matched is greater than or equal to a second preset threshold, it is determined that there is a common view relationship between the current image frame and the current sub-map, and step 203 is performed. Otherwise, it is determined that the current image frame and the current sub-map do not have the co-view relationship, step 205 is executed.
In step 203, the current sub-map is extended with the current image frame.
In step 204, in response to the current image frame and at least one other sub-map having a co-view relationship, the current sub-map and at least one other sub-map are merged.
In step 205, a new sub-map is created with empty content.
In step 206, in the new sub-map with empty content, the current image frame is used as the initial frame of the new sub-map.
In step 207, the new current image frame is visually tracked in combination with the new sub-map, and a new visual tracking result is obtained.
In step 208, it is determined whether the new current image frame and the new sub-map have a co-view relationship according to the new visual tracking result.
In step 209, in response to the new current image frame and the new sub-map having a co-view relationship, the new current sub-map is expanded by the new current image frame, otherwise, the new sub-map is created again.
In step 210, all the sub-maps are merged to obtain a global map.
And in response to the common-view relationship between the at least two sub-maps, combining the sub-maps with the common-view relationship by adopting a loop closing algorithm, and eliminating errors through optimization to obtain the global map.
In the above embodiment, the sub-map expansion and the sub-map generation are performed based on whether the current image frame and the current sub-map have a common view relationship, so as to solve the problem that the map cannot be expanded once the vision in the single map is difficult to continue tracking. Corresponding to the foregoing method embodiments, the present disclosure also provides embodiments of an apparatus.
As shown in fig. 8, fig. 8 is a block diagram of a map building apparatus shown in accordance with an exemplary embodiment of the present disclosure, the apparatus comprising: the first visual tracking module 310 is configured to perform visual tracking on the current image frame in combination with the current sub-map to obtain a visual tracking result; a first co-view relationship determining module 320, configured to determine whether a co-view relationship exists between the current image frame and the current sub-map according to the visual tracking result; the first map building module 330 is configured to create a new sub-map in response to no co-view relationship between the current image frame and the current sub-map.
Fig. 9 is a block diagram of another map expanding apparatus shown in fig. 9 according to an exemplary embodiment of the present disclosure, where on the basis of the foregoing fig. 8 embodiment, the first common view relationship determining module 320 includes: and a first determining sub-module 321, configured to determine that the current image frame and the current sub-map have a co-view relationship in response to the visual tracking result indicating that there are matched feature points between the current image frame and the current sub-map, and the number of matched feature points is greater than or equal to a first preset threshold, otherwise, determine that the current image frame and the current sub-map do not have a co-view relationship.
Fig. 10 is a block diagram of another mapping apparatus according to an exemplary embodiment of the present disclosure, which is based on the foregoing fig. 9 embodiment, and the first co-view relationship determining module 320 includes: the obtaining sub-module 322 is configured to, in response to the visual tracking result indicating that there are matched feature points between the current image frame and the current sub-map, remove a first feature point that is a wrong match from the matched feature points, and obtain a second feature point that is a valid match; and the second determining sub-module 323 is configured to determine that the current image frame and the current sub-map have a co-view relationship in response to the number of the second feature points being greater than or equal to a second preset threshold, and otherwise determine that the current image frame and the current sub-map do not have a co-view relationship.
Fig. 11 is a block diagram of another mapping apparatus according to an exemplary embodiment of the present disclosure, as shown in fig. 11, where on the basis of the foregoing fig. 10 embodiment, the obtaining sub-module 322 includes: an obtaining unit 3221 is configured to input the current image frame and the image frame included in the current sub-map into a feature point matching model established in advance, and remove the first feature point in the matching feature point through the feature point matching model, so as to obtain the second feature point in effective matching.
Fig. 12 is a block diagram of another mapping apparatus according to an exemplary embodiment of the present disclosure, as shown in fig. 12, which is based on the foregoing fig. 8 embodiment, and the apparatus further includes: a second map construction module 340, configured to expand the current sub-map using the current image frame in response to a co-view relationship between the current image frame and the current sub-map.
Fig. 13 is a block diagram of another mapping apparatus according to an exemplary embodiment of the present disclosure, as shown in fig. 13, which is based on the foregoing fig. 8 embodiment, and the apparatus further includes: a third map building module 350, configured to, in response to a co-view relationship between the current image frame and at least one of the other sub-maps, merge the current sub-map with at least one of the other sub-maps.
Fig. 14 is a block diagram of another mapping apparatus according to an exemplary embodiment of the present disclosure, as shown in fig. 14, based on the foregoing fig. 8 embodiment, where the first mapping module 330 includes: a first map building sub-module 331 for creating a new sub-map whose contents are empty; a second map construction sub-module 332, configured to use the current image frame as an initial frame of the new sub-map in the new sub-map with empty content.
Fig. 15 is a block diagram of another mapping apparatus according to an exemplary embodiment of the present disclosure, as shown in fig. 15, which is based on the foregoing fig. 8 embodiment, and the apparatus further includes: the second visual tracking module 360 is configured to perform visual tracking on the new current image frame in combination with the new sub-map to obtain a new visual tracking result; a second co-view relationship determining module 370, configured to determine whether a co-view relationship exists between the new current image frame and the new sub-map according to the new visual tracking result; a fourth map building module 380, configured to create a new sub-map again in response to the new current image frame and the new sub-map having no co-view relationship.
Fig. 16 is a block diagram of another mapping apparatus according to an exemplary embodiment of the present disclosure, as shown in fig. 16, which is based on the foregoing fig. 8 embodiment, and the apparatus further includes: and a fifth map building module 390, configured to combine all sub-maps that meet a preset combining condition to obtain a global map.
Fig. 17 shows another block diagram of a mapping apparatus according to an exemplary embodiment of the present disclosure, which is based on the foregoing fig. 16 embodiment, and the fifth mapping module 390 includes: and a third determining sub-module 391, configured to, in response to a co-view relationship between at least two sub-maps, determine that the at least two sub-maps with the co-view relationship are sub-maps meeting a preset merging condition.
For the device embodiments, since they substantially correspond to the method embodiments, reference may be made to the partial description of the method embodiments for relevant points. The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the disclosed solution. One of ordinary skill in the art can understand and implement it without inventive effort.
The embodiment of the disclosure also provides a computer-readable storage medium, which stores a computer program for executing any one of the map building methods.
In some optional embodiments, the disclosed embodiments provide a computer program product comprising computer readable code which, when run on a device, a processor in the device executes instructions for implementing a mapping method as provided in any of the above embodiments.
In some optional embodiments, the present disclosure further provides another computer program product for storing computer readable instructions, which when executed, cause a computer to perform the operations of the map building method provided in any one of the above embodiments.
The computer program product may be embodied in hardware, software or a combination thereof. In an alternative embodiment, the computer program product is embodied in a computer storage medium (including volatile and non-volatile storage media), and in another alternative embodiment, the computer program product is embodied in a Software product, such as a Software Development Kit (SDK), or the like.
The embodiment of the present disclosure further provides a map building apparatus, which includes: a processor; a memory for storing processor-executable instructions; the processor is used for calling the executable instructions stored in the memory to realize any map construction method.
As shown in fig. 18, fig. 18 is a schematic diagram illustrating a structure of a map building apparatus 1800, according to an exemplary embodiment. For example, apparatus 1800 may be provided as a machine device, terminal, or platform. Referring to fig. 18, the apparatus 1800 includes a processing component 1822 that further includes one or more processors and memory resources, represented by memory 1832, for storing instructions, such as applications, that are executable by the processing component 1822. The application programs stored in memory 1832 may include one or more modules that each correspond to a set of instructions. Further, the processing component 1822 is configured to execute instructions to perform any of the mapping methods described above.
The device 1800 may also include a power component 1826 for performing power management of the device 1800, a wired or wireless network interface 1850 for connecting the device 1800 to a network, and an input-output (I/O) interface 1858. The apparatus 1800 may operate based on an operating system stored in the memory 1832, such as Windows Server, Mac OS XTM, UnixTM, LinuxTM, FreeB SDTM, or the like.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
The above description is only exemplary of the present disclosure and should not be taken as limiting the disclosure, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present disclosure should be included in the scope of the present disclosure.
Claims (10)
1. A map construction method, characterized in that the method comprises:
performing visual tracking on the current image frame in combination with the current sub-map to obtain a visual tracking result;
determining whether the current image frame and the current sub map have a common-view relationship according to the visual tracking result;
and in response to the current image frame and the current sub-map having no co-view relationship, creating a new sub-map.
2. The method of claim 1, wherein the determining whether the current image frame and the current sub-map have a co-view relationship according to the visual tracking result comprises:
and in response to the visual tracking result indicating that matched feature points exist between the current image frame and the current sub-map and the number of matched feature points is greater than or equal to a first preset threshold value, determining that a common-view relationship exists between the current image frame and the current sub-map, otherwise, determining that the common-view relationship does not exist between the current image frame and the current sub-map.
3. The method of claim 1, wherein the determining whether the current image frame and the current sub-map have a co-view relationship according to the visual tracking result comprises:
in response to the visual tracking result indicating that there are matched feature points between the current image frame and the current sub-map, removing a first feature point which is in a wrong match from the matched feature points, and obtaining a second feature point which is in a valid match;
and in response to the number of the second characteristic points being greater than or equal to a second preset threshold, determining that the current image frame and the current sub-map have the common-view relationship, otherwise determining that the current image frame and the current sub-map do not have the common-view relationship.
4. The method according to claim 3, wherein obtaining a valid matching second feature point after removing a first feature point which is in a wrong matching from the matched feature points comprises:
inputting the current image frame and the current sub map into a pre-established feature point matching model, and removing the first feature points which are in error matching from the matched feature points through the feature point matching model to obtain the second feature points which are in effective matching.
5. A map building apparatus, characterized in that the apparatus comprises:
the first visual tracking module is used for carrying out visual tracking on the current image frame in combination with the current sub-map to obtain a visual tracking result;
the first co-view relationship determining module is used for determining whether the current image frame and the current sub-map have a co-view relationship according to the visual tracking result;
and the first map building module is used for responding to the condition that the current image frame and the current sub-map do not have the common view relation, and then creating a new sub-map.
6. The apparatus of claim 5, wherein the first co-view relationship determining module comprises:
a first determining sub-module, configured to determine that there is a common-view relationship between the current image frame and the current sub-map if the visual tracking result indicates that there are matched feature points between the current image frame and the current sub-map and the number of matched feature points is greater than or equal to a first preset threshold, and otherwise determine that there is no common-view relationship between the current image frame and the current sub-map.
7. The apparatus of claim 5, wherein the first co-view relationship determining module comprises:
the obtaining sub-module is used for responding to the visual tracking result to indicate that matched feature points exist between the current image frame and the current sub-map, removing first feature points which are in error matching from the matched feature points, and obtaining second feature points which are in effective matching;
and the second determining sub-module is used for determining that the current image frame and the current sub-map have a common-view relationship in response to the number of the second characteristic points being greater than or equal to a second preset threshold, and otherwise determining that the current image frame and the current sub-map do not have the common-view relationship.
8. The apparatus of claim 7, wherein the acquisition submodule comprises:
and the obtaining unit is used for inputting the current image frame and the image frame included in the current sub-map into a pre-established feature point matching model, and removing the first feature points which are in error matching from the matched feature points through the feature point matching model to obtain the second feature points which are effectively matched.
9. A computer-readable storage medium, characterized in that the storage medium stores a computer program for executing the map construction method of any of the above claims 1 to 4.
10. A multi-map maintenance apparatus, the apparatus comprising:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to invoke executable instructions stored in the memory to implement the mapping method of any of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910944288.0A CN112581514A (en) | 2019-09-30 | 2019-09-30 | Map construction method and device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910944288.0A CN112581514A (en) | 2019-09-30 | 2019-09-30 | Map construction method and device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112581514A true CN112581514A (en) | 2021-03-30 |
Family
ID=75116680
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910944288.0A Pending CN112581514A (en) | 2019-09-30 | 2019-09-30 | Map construction method and device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112581514A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113984072A (en) * | 2021-10-28 | 2022-01-28 | 阿波罗智能技术(北京)有限公司 | Vehicle positioning method, device, equipment, storage medium and automatic driving vehicle |
| CN113984072B (en) * | 2021-10-28 | 2024-05-17 | 阿波罗智能技术(北京)有限公司 | Vehicle positioning method, device, equipment, storage medium and automatic driving vehicle |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105953796A (en) * | 2016-05-23 | 2016-09-21 | 北京暴风魔镜科技有限公司 | Stable motion tracking method and stable motion tracking device based on integration of simple camera and IMU (inertial measurement unit) of smart cellphone |
| WO2017172778A1 (en) * | 2016-03-28 | 2017-10-05 | Sri International | Collaborative navigation and mapping |
| CN108319976A (en) * | 2018-01-25 | 2018-07-24 | 北京三快在线科技有限公司 | Build drawing method and device |
| CN108665540A (en) * | 2018-03-16 | 2018-10-16 | 浙江工业大学 | Robot localization based on binocular vision feature and IMU information and map structuring system |
| CN109059941A (en) * | 2018-07-06 | 2018-12-21 | 禾多科技（北京）有限公司 | Characteristics map construction method, vision positioning method and corresponding intrument |
| CN109631855A (en) * | 2019-01-25 | 2019-04-16 | 西安电子科技大学 | High-precision vehicle positioning method based on ORB-SLAM |
| CN109816696A (en) * | 2019-02-01 | 2019-05-28 | 西安全志科技有限公司 | A kind of robot localization and build drawing method, computer installation and computer readable storage medium |
| CN109887053A (en) * | 2019-02-01 | 2019-06-14 | 广州小鹏汽车科技有限公司 | A kind of SLAM map joining method and system |
| WO2019169540A1 (en) * | 2018-03-06 | 2019-09-12 | 斯坦德机器人（深圳）有限公司 | Method for tightly-coupling visual slam, terminal and computer readable storage medium |
-
2019
- 2019-09-30 CN CN201910944288.0A patent/CN112581514A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017172778A1 (en) * | 2016-03-28 | 2017-10-05 | Sri International | Collaborative navigation and mapping |
| CN105953796A (en) * | 2016-05-23 | 2016-09-21 | 北京暴风魔镜科技有限公司 | Stable motion tracking method and stable motion tracking device based on integration of simple camera and IMU (inertial measurement unit) of smart cellphone |
| CN108319976A (en) * | 2018-01-25 | 2018-07-24 | 北京三快在线科技有限公司 | Build drawing method and device |
| WO2019169540A1 (en) * | 2018-03-06 | 2019-09-12 | 斯坦德机器人（深圳）有限公司 | Method for tightly-coupling visual slam, terminal and computer readable storage medium |
| CN108665540A (en) * | 2018-03-16 | 2018-10-16 | 浙江工业大学 | Robot localization based on binocular vision feature and IMU information and map structuring system |
| CN109059941A (en) * | 2018-07-06 | 2018-12-21 | 禾多科技（北京）有限公司 | Characteristics map construction method, vision positioning method and corresponding intrument |
| CN109631855A (en) * | 2019-01-25 | 2019-04-16 | 西安电子科技大学 | High-precision vehicle positioning method based on ORB-SLAM |
| CN109816696A (en) * | 2019-02-01 | 2019-05-28 | 西安全志科技有限公司 | A kind of robot localization and build drawing method, computer installation and computer readable storage medium |
| CN109887053A (en) * | 2019-02-01 | 2019-06-14 | 广州小鹏汽车科技有限公司 | A kind of SLAM map joining method and system |
Non-Patent Citations (1)
| Title |
|---|
| 张剑华 等: "单目同时定位与建图中的地图恢复融合技术", 《中国图象图形学报》, vol. 23, no. 3, 31 December 2018 (2018-12-31), pages 372 - 383 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113984072A (en) * | 2021-10-28 | 2022-01-28 | 阿波罗智能技术(北京)有限公司 | Vehicle positioning method, device, equipment, storage medium and automatic driving vehicle |
| CN113984072B (en) * | 2021-10-28 | 2024-05-17 | 阿波罗智能技术(北京)有限公司 | Vehicle positioning method, device, equipment, storage medium and automatic driving vehicle |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107990899B (en) | Positioning method and system based on SLAM | |
| CN111862296B (en) | Three-dimensional reconstruction method, three-dimensional reconstruction device, three-dimensional reconstruction system, model training method and storage medium | |
| CN107845114B (en) | Map construction method and device and electronic equipment | |
| CN111445526B (en) | Method, device and storage medium for estimating pose of image frame | |
| US20210019542A1 (en) | Multi-Angle Object Recognition | |
| WO2020221307A1 (en) | Method and device for tracking moving object | |
| US20230316690A1 (en) | 3-D Reconstruction Using Augmented Reality Frameworks | |
| JP2021534600A (en) | Video stabilization methods and equipment, as well as non-temporary computer-readable media | |
| US20190147601A1 (en) | Information processing apparatus, background image update method, and non-transitory computer-readable storage medium | |
| US11176700B2 (en) | Systems and methods for a real-time intelligent inspection assistant | |
| CN110956646A (en) | Target tracking method, device, equipment and storage medium | |
| CN111016887A (en) | Automatic parking device and method for motor vehicle | |
| CN113112542A (en) | Visual positioning method and device, electronic equipment and storage medium | |
| CN114913182A (en) | Image segmentation method, device, equipment and storage medium | |
| JP2014238674A (en) | Information processing system, information processing method, and program | |
| CN112556699B (en) | Navigation positioning method and device, electronic equipment and readable storage medium | |
| CN110470296A (en) | A kind of localization method, positioning robot and computer storage medium | |
| US11188754B2 (en) | Method for improving robustness of visual-inertial navigation system, and robot thereof | |
| CN112581514A (en) | Map construction method and device and storage medium | |
| CN111950517A (en) | Target detection method, model training method, electronic device and storage medium | |
| CN112233149A (en) | Scene flow determination method and device, storage medium and electronic device | |
| CN113591885A (en) | Target detection model training method, device and computer storage medium | |
| CN111862150A (en) | Image tracking method and device, AR device and computer device | |
| CN114299192B (en) | Method, device, equipment and medium for positioning and mapping | |
| CN114415698B (en) | Robot, positioning method and device of robot and computer equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |