WO2020259361A1 - 地图更新方法及装置、终端、存储介质 - Google Patents

地图更新方法及装置、终端、存储介质 Download PDF

Info

Publication number
WO2020259361A1
WO2020259361A1 PCT/CN2020/096505 CN2020096505W WO2020259361A1 WO 2020259361 A1 WO2020259361 A1 WO 2020259361A1 CN 2020096505 W CN2020096505 W CN 2020096505W WO 2020259361 A1 WO2020259361 A1 WO 2020259361A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
map
key frame
feature
frame image
Prior art date
Application number
PCT/CN2020/096505
Other languages
English (en)
French (fr)
Inventor
金珂
杨宇尘
陈岩
方攀
Original Assignee
Oppo广东移动通信有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Oppo广东移动通信有限公司 filed Critical Oppo广东移动通信有限公司
Priority to EP20831138.1A priority Critical patent/EP3975003A4/en
Publication of WO2020259361A1 publication Critical patent/WO2020259361A1/zh
Priority to US17/549,840 priority patent/US20220099455A1/en

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23Updating
    • GPHYSICS
    • G01MEASURING; TESTING
    • G01CMEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/38Electronic maps specially adapted for navigation; Updating thereof
    • G01C21/3804Creation or updating of map data
    • G01C21/3833Creation or updating of map data characterised by the source of data
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/29Geographical information databases
    • GPHYSICS
    • G01MEASURING; TESTING
    • G01CMEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/38Electronic maps specially adapted for navigation; Updating thereof
    • G01C21/3863Structures of map data
    • G01C21/387Organisation of map data, e.g. version management or database structures
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/587Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using geographical or spatial information, e.g. location
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/40Extraction of image or video features
    • G06V10/44Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00Scenes; Scene-specific elements
    • G06V20/40Scenes; Scene-specific elements in video content
    • G06V20/46Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00Scenes; Scene-specific elements
    • G06V20/50Context or environment of the image
    • G06V20/56Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle

Definitions

  • This application relates to indoor positioning technology, but is not limited to map update methods and devices, terminals, and storage media.
  • the global map is constructed by the histogram correlation of the two local maps, but the calculation of the angle offset and translation offset of the histogram is dependent on the normal phase characteristics of the point cloud; in this way, due to the normal phase characteristics of the point cloud The accuracy is not high, and errors are prone to occur, which will result in low accuracy of the obtained map.
  • the embodiments of the present application provide a map update method and device, terminal, and storage medium in order to solve at least one problem in the related art.
  • the embodiment of the present application provides a map update method, the method includes:
  • a second image feature is matched from a second map; wherein the scene information corresponding to the first map and the second map are the same;
  • each key frame image in the first map is added to the second map to obtain an updated second map.
  • the acquiring the first image feature in the first map includes:
  • the matching of the second image feature from the second map according to the first image feature includes:
  • the second image feature is matched from the image feature of the key frame image in the second map.
  • adding each key frame image in the first map to the second map according to the first image feature and the second image feature to obtain an updated second map includes:
  • each key frame image in the first map is added to the second map to obtain an updated second map.
  • adding each key frame image in the first map to the second map according to the first location information and the second location information to obtain an updated second map include:
  • the coordinates of the image acquisition device corresponding to each key frame image in the first map are adjusted in the first coordinate system, so that the adjustment of the image acquisition device corresponding to each key frame image
  • the last coordinate matches the second coordinate system
  • the key frame image corresponding to the image acquisition device after each coordinate adjustment is added to the second map to obtain the updated second map.
  • the scene information corresponding to the first map and the second map are at least partially the same.
  • the first image feature includes: two-dimensional (2-Dimensional, 2D) location information, (3-Dimensional, 3D) location information and identification of the feature points of the key frame image in the first map information;
  • the second image feature includes: two-dimensional location information, three-dimensional location information, and identification information of the feature points of the key frame image in the second map; wherein, the three-dimensional location information maps the two-dimensional location information Obtained in the coordinate system where the two-dimensional position information is located.
  • extracting the key frame image in the first map to obtain the first image feature includes:
  • Each two-dimensional location information is respectively mapped in the coordinate system where the first map is located to obtain the three-dimensional location information of each feature point.
  • matching the second image feature from the image features of the key frame image in the second map according to the first image feature includes:
  • the second ratio vector is the ratio of the multiple sample feature points among the feature points included in the key frame image in the second map
  • a second image feature is matched from the image features of the key frame image in the second map.
  • the second image is matched from the image features of the key frame images in the second map according to the first image feature, the first ratio vector, and the second ratio vector Features include:
  • the first ratio vector and the second ratio vector from the image features of the key frame images in the second map, determine similar image features whose similarity to the first image feature is greater than a second threshold ;
  • a second image feature whose similarity with the first image feature meets a preset similarity threshold is selected.
  • the selecting a second image feature whose similarity with the first image feature meets a preset similarity threshold from the image features of the similar key frame images includes:
  • a second image feature whose similarity with the first image feature meets a preset similarity threshold is selected.
  • the selecting a second image feature whose similarity with the first image feature meets a preset similarity threshold from the image features of the joint frame image includes:
  • the identification information of the feature points of the target joint frame image and the identification information of the feature points of the key frame image to be matched from the image features of the target joint frame image, select the feature similarity with the first image to meet the predetermined Set the second image feature of similarity threshold.
  • the method before determining the second position information of the image acquisition device in the second coordinate system where the second map is located based on the first image feature and the second image feature, the method also includes:
  • the determining, according to the first image feature and the second image feature, the second position information of the image acquisition device in the second coordinate system where the second map is located includes:
  • the number of target Euclidean distances included in the target Euclidean distance set is greater than the fifth threshold, based on the three-dimensional position information of the feature points of the key frame image corresponding to the second image feature and the key frame corresponding to the first image feature The two-dimensional position information of the feature points of the image determines the second position information.
  • the method before extracting the key frame image in the first map to obtain the first image feature, the method further includes:
  • the method before selecting key frame images meeting preset conditions from the sample image library to obtain the key frame image set, the method further includes:
  • the sixth threshold it is determined that the scene corresponding to the sample image is a discrete scene.
  • the scene corresponding to the sample image is a discrete scene, select the key frame image from the sample image library according to the input selection instruction;
  • the key frame image is selected from the sample image library according to the preset frame rate or disparity.
  • the determining the ratio of each sample feature point in the key frame image to obtain the ratio vector set includes:
  • the first average number is determined; where i is an integer greater than or equal to 1; The first average number is used to indicate the average number of times the i-th sample feature point appears in each sample image;
  • the ratio of the sample feature point in the key frame image is obtained, and the ratio vector set is obtained.
  • An embodiment of the application provides a map update device, the device includes: a first acquisition module, a first matching module, and a first update module, wherein:
  • the first acquiring module is configured to acquire the first image feature in the first map
  • the first matching module is configured to match a second image feature from a second map according to the first image feature; wherein the scene information parts corresponding to the first map and the second map are the same;
  • the first update module is configured to add each key frame image in the first map to the second map according to the first image feature and the second image feature to obtain an updated second map.
  • the first obtaining module includes:
  • the first extraction sub-module is configured to extract the key frame image to be matched in the first map to obtain the first image feature; wherein the key frame image to be matched is the key frame image in the first map;
  • the first matching module includes:
  • the first matching sub-module is configured to match the second image feature from the image feature of the key frame image in the second map according to the first image feature.
  • the first update module includes:
  • a first acquisition submodule configured to acquire first position information of an image acquisition device used to acquire the key frame image to be matched in a first coordinate system where the first map is located;
  • a first determining sub-module configured to determine, according to the first image feature and the second image feature, second location information of the image acquisition device in a second coordinate system where the second map is located;
  • the first update submodule is configured to add each key frame image in the first map to the second map according to the first location information and the second location information to obtain an updated second map .
  • the first update submodule includes:
  • a first determining unit configured to determine a conversion relationship between the first coordinate system and the second coordinate system according to the first position information and the second position information;
  • the first adjustment unit is configured to adjust the coordinates in the first coordinate system of the image acquisition device corresponding to each key frame image in the first map according to the conversion relationship, so that each key frame image The adjusted coordinates of the corresponding image acquisition device match the second coordinate system;
  • the first adding unit is configured to add the key frame image corresponding to the image acquisition device after each coordinate adjustment to the second map to obtain the updated second map.
  • the scene information corresponding to the first map and the second map are at least partially the same.
  • the first image feature includes: two-dimensional location information, three-dimensional location information, and identification information of feature points of the key frame image in the first map;
  • the second image feature includes: two-dimensional location information, three-dimensional location information, and identification information of the feature points of the key frame image in the second map; wherein, the three-dimensional location information maps the two-dimensional location information Obtained in the coordinate system where the two-dimensional position information is located.
  • the first extraction submodule includes:
  • the first extraction unit is configured to extract feature points of each key frame image in the first map to obtain a feature point set
  • the second determining unit is configured to determine the identification information of each feature point in the feature point set and the two-dimensional position information of each feature point in the key frame image;
  • the first mapping unit is configured to map each two-dimensional location information to the coordinate system where the first map is located to obtain the three-dimensional location information of each feature point.
  • the first matching submodule includes:
  • the third determining unit is configured to separately determine the ratios of different sample feature points in the feature point set to obtain a first ratio vector
  • the first obtaining unit is configured to obtain a second ratio vector, where the second ratio vector is the ratio of the plurality of sample feature points among the feature points contained in the key frame image in the second map;
  • the first matching unit is configured to match the second image from the image features of the key frame images in the second map according to the first image feature, the first ratio vector, and the second ratio vector feature.
  • the first matching unit includes:
  • the first determining subunit is configured to determine the similarity to the first image feature from the image features of the key frame image in the second map according to the first ratio vector and the second ratio vector Similar image features greater than the second threshold;
  • the second determining subunit is configured to determine similar key frame images to which the similar image features belong to obtain a set of similar key frame images
  • the first selection subunit is configured to select, from the image features of the similar key frame images, a second image feature whose similarity with the first image feature meets a preset similarity threshold.
  • the first selection subunit is configured to: determine the time difference between the acquisition times of at least two similar key frame images, and the image characteristics of the at least two similar key frame images are respectively compared with all The similarity difference of the first image feature; the similar key frame images whose time difference is smaller than the third threshold and the similarity difference is smaller than the fourth threshold are combined to obtain a joint frame image; from the image of the joint frame image Among the features, a second image feature whose similarity with the first image feature meets a preset similarity threshold is selected.
  • the first selection subunit is configured to: respectively determine the sum of similarity between the image feature of each key frame image contained in the multiple joint frame images and the first image feature;
  • the joint frame image with the largest sum is determined as the target joint frame image with the highest similarity to the key frame image to be matched; according to the identification information of the characteristic points of the target joint frame image and the characteristic points of the key frame image to be matched
  • the identification information is to select a second image feature whose similarity with the first image feature meets a preset similarity threshold from the image features of the target joint frame image.
  • the device further includes:
  • the first determining module is configured to determine a target Euclidean distance between any two feature points included in the key frame image corresponding to the second image feature that is less than a first threshold to obtain a target Euclidean distance set;
  • the first determining submodule includes:
  • the fourth determining unit is configured to, if the number of target Euclidean distances included in the target Euclidean distance set is greater than a fifth threshold, based on the three-dimensional position information of the feature point of the key frame image corresponding to the second image feature and the first The two-dimensional location information of the feature point of the key frame image corresponding to an image feature determines the second location information.
  • the device further includes:
  • the first selection module is configured to select key frame images meeting preset conditions from the sample image library to obtain a set of key frame images
  • the first extraction module is configured to extract the image features of each key frame image to obtain a key image feature set
  • the second extraction module is configured to extract feature points of the sample image to obtain a sample feature point set containing different feature points
  • the second determining module is configured to determine the ratio of each sample feature point in the key frame image to obtain a set of ratio vectors
  • the first storage module is configured to store the ratio vector set and the key image feature set to obtain the first map.
  • the device further includes:
  • the second selection module is configured to select a preset number of corner points from the sample image
  • the third determining module is configured to determine that the scene corresponding to the sample image is a continuous scene if the number of the same corner points contained in two sample images adjacent to the acquisition time is greater than or equal to a sixth threshold;
  • the fourth determining module is configured to determine that the scene corresponding to the sample image is a discrete scene if the number of the same corner points contained in the two sample images adjacent to the acquisition time is less than the sixth threshold.
  • the first selection module includes:
  • the first selection submodule is configured to, if the scene corresponding to the sample image is a discrete scene, select the key frame image from the sample image library according to the input selection instruction;
  • the second selection sub-module is configured to select a key frame image from the sample image library according to a preset frame rate or disparity if the scene corresponding to the sample image is a continuous scene.
  • the second determining module includes:
  • the second determining submodule is configured to determine the first average number of times according to the first number of sample images contained in the sample image library and the first number of times the i-th sample feature point appears in the sample image library; where i is An integer greater than or equal to 1; the first average number is configured to indicate the average number of times the i-th sample feature point appears in each sample image;
  • the third determining submodule is configured to, based on the second number of occurrences of the i-th sample feature point in the j-th key frame image and the second number of sample feature points contained in the j-th key frame image, Determine the second average number; where j is an integer greater than or equal to 1; the second average number is configured to indicate the ratio of the i-th sample feature point to the sample feature points contained in the j-th key frame image;
  • the fourth determining submodule is configured to obtain the ratio of the sample feature points in the key frame image according to the first average number and the second average number, and obtain the ratio vector set.
  • An embodiment of the present application provides a terminal including a memory and a processor, the memory stores a computer program that can run on the processor, and the processor implements the steps in the above map update method when the program is executed.
  • the embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above map update method are implemented.
  • the embodiments of the present application provide a map update method, device, terminal, and storage medium.
  • First, the first image feature in the first map is acquired; then, the first image feature is matched from the second map according to the first image feature.
  • the key frame image corresponding to the second image feature can be obtained. Therefore, based on the matching image features in different maps, multiple maps are merged, thereby realizing the update of the map and improving the accuracy of the updated map.
  • 1A is a schematic diagram of the implementation process of the map update method according to the embodiment of the application.
  • FIG. 1B is a schematic diagram of another implementation process of the map update method according to the embodiment of this application.
  • FIG. 2A is a schematic diagram of the implementation process of the map update method according to the embodiment of this application.
  • 2B is a schematic diagram of another implementation process of creating a preset map according to an embodiment of the application.
  • 2C is a schematic diagram of another implementation process of the map update method according to the embodiment of this application.
  • FIG. 3 is a schematic diagram of another implementation process of the map update method according to the embodiment of the application.
  • FIG. 4 is a schematic diagram of the structure of a ratio vector in an embodiment of the application.
  • FIG. 5A is an application scene diagram of determining a key frame image corresponding to a second image feature according to an embodiment of the application
  • 5B is a schematic structural diagram of determining location information of a collection device according to an embodiment of the application.
  • FIG. 6 is a schematic diagram of the composition structure of a map updating apparatus according to an embodiment of the application.
  • FIG. 1A is a schematic diagram of the implementation process of the map update method according to the embodiment of the application. As shown in FIG. 1A, the method includes the following steps:
  • Step S101 Acquire a first image feature in a first map.
  • the first map may be understood as a part of the global map.
  • the global map contains key frame images corresponding to 100 indoor scenes
  • the first map may contain key frame images corresponding to part of the indoor scenes.
  • the first image feature includes: 2D location information, 3D location information, and identification information of the feature points of the key frame image in the first map.
  • step S101 firstly, extract the feature points of each key frame image in the first map to obtain a feature point set; then, determine the identification information of each feature point in the feature point set and each The 2D location information of the feature point in the key frame image; wherein the identification information of the feature point can be understood as the descriptor information that can uniquely identify the feature point; finally, each 2D location information is mapped to the first In the coordinate system where a map is located, the 3D position information of each of the feature points is obtained.
  • Step S102 matching the second image feature from the second map according to the first image feature.
  • the first map and the second map are two different maps, and the scene information parts corresponding to the first map and the second map are the same, so as to ensure that the key frames in the first map can be matched from the second map.
  • the first image feature of the image is similar to the second image feature; the second map can also be understood as a part of the global map.
  • the global map contains key frame images corresponding to 100 indoor scenes.
  • the second map may contain key frame images corresponding to a part of the indoor scene.
  • the scene information corresponding to the first map and the second map are at least partially the same; that is, there is a slight overlap between the first map and the second map.
  • the second image feature includes: the second image feature includes: 2D location information, 3D location information, and identification information of the feature points of the key frame image in the second map; wherein, the 3D location information is The 2D position information is obtained by mapping in the coordinate system where the 2D position information is located.
  • the step S102 can be understood as selecting a second image feature with a higher degree of matching with the first image feature from the image features of the key frame image stored in the second map.
  • Step S103 Add each key frame image in the first map to the second map according to the first image feature and the second image feature to obtain an updated second map.
  • the key frame corresponding to the first image feature is determined based on the 3D location information of the feature point of the key frame image corresponding to the second image feature and the 2D location information of the feature point of the key frame image corresponding to the first image feature Image location information of the image capture device. For example, first, the 2D location information of the feature points of the key frame image corresponding to the first image feature is converted into 3D location information, and then the 3D location information is combined with the key frame image in the second coordinate system where the second map is located The 3D location information of the feature points is compared to determine the location information of the image acquisition device of the key frame image corresponding to the first image feature.
  • the image acquisition device for the key frame image to be matched corresponding to the first image feature can be obtained.
  • the 2D position information of the image capture device can also be obtained from the 3D position information of the image capture device of the key frame image corresponding to the first image feature. It can also be understood as not only the planar space position of the image capture device image, but also the image capture device's Three-dimensional spatial location; in this way, based on the rich location information, multiple local maps can be merged more accurately.
  • any key frame image in the first map by extracting image features, firstly, find the second image feature matching the image feature from the second map, and secondly, based on the two image features Finally, based on the conversion relationship between the location information of the image features and the coordinate systems corresponding to the two maps, each key frame image in the first map is added to the second map, thereby completing The update of the second map ensures good map fusion accuracy.
  • FIG. 1B is a schematic diagram of another implementation process of the map update method according to the embodiment of the application. As shown in FIG. 1B, the method includes the following steps:
  • Step S121 Extract the key frame image to be matched in the first map to obtain the first image feature.
  • the key frame image to be matched is the key frame image in the first map.
  • Step S122 According to the first image feature, the second image feature is matched from the image feature of the key frame image in the second map.
  • a preset bag-of-words model is used to retrieve a second image feature with a higher similarity to the first image feature from the image features of the key frame image stored in the second map.
  • Step S123 Acquire first position information of an image acquisition device used to acquire the key frame image to be matched in the first coordinate system where the first map is located.
  • the first coordinate system may be a three-dimensional coordinate system
  • the first position information may be understood as the three-dimensional coordinate value in the first coordinate system of the image acquisition device to be matched with the key frame image.
  • Step S124 According to the first image feature and the second image feature, determine second location information of the image acquisition device in the second coordinate system where the second map is located.
  • the second coordinate system is a three-dimensional coordinate system
  • the second coordinate system is a three-dimensional coordinate system
  • the 2D location information of the feature point of the key frame image corresponding to the image feature determines the second location information.
  • the second position information can be understood as the three-dimensional coordinate value of the image acquisition device to be matched with the key frame image in the second coordinate system.
  • the 3D position information of the feature point of the key frame image corresponding to the second image feature and the 2D position information of the feature point corresponding to the first image feature are used as a front-end pose tracking algorithm (Perspectives-n-Point, PnP) Algorithm input, first obtain the 2D position information (for example, 2D coordinates) of the feature point in the key frame image to be matched in the first coordinate system (for example, 3D coordinates), and then according to the second coordinate
  • the 3D position information of the feature point corresponding to the second feature and the 3D position information of the feature point in the key frame image to be matched in the first coordinate system can be used to solve the position information of the image acquisition device of the key frame image to be matched.
  • Step S125 Add each key frame image in the first map to the second map according to the first location information and the second location information to obtain an updated second map.
  • the updated second map may be a global map or a part of the global map.
  • the step S125 can be implemented through the following process:
  • the first step is to determine the conversion relationship between the first coordinate system and the second coordinate system according to the first position information and the second position information.
  • the conversion relationship between the first coordinate system and the second coordinate system may be the rotation matrix and translation vector of the first coordinate system relative to the second coordinate system.
  • the second step is to adjust the coordinates in the first coordinate system of the image acquisition device corresponding to each key frame image in the first map according to the conversion relationship, so that the image corresponding to each key frame image
  • the adjusted coordinates of the acquisition device match the second coordinate system.
  • the process of adjusting the coordinates in the first coordinate system of the image acquisition device corresponding to each key frame image in the first map may be: first, using a rotation matrix to adjust each The coordinates of the image capture device corresponding to the key frame image in the first coordinate system are rotated; then the translation vector is used to translate the rotated coordinates of the image capture device corresponding to each key frame image in the first coordinate system. , Obtain the adjusted coordinates of the image acquisition device corresponding to each key frame image.
  • the key frame image corresponding to the image acquisition device after each coordinate adjustment is added to the second map to obtain the updated second map.
  • the coordinates of the image capture device corresponding to each key frame image in the first coordinate system have been adjusted to the coordinates matching the second coordinate system, it is possible to change each key frame in the first map based on this
  • the image is added to the second map to realize the fusion of the two maps to obtain an updated second map (ie, the global map).
  • three or more maps may be merged to obtain a global map.
  • steps S123 to S125 give an implementation "according to the first image feature and the second image feature, add each key frame image in the first map to the second map to obtain “Updated second map” method, in which, based on the position information of the image capture device to be matched with the key frame in the first coordinate system and the second coordinate system, the first coordinate system and the second coordinate system are determined.
  • multiple local maps are merged based on the image features of the key frame images in the two layout maps to achieve the purpose of map update, so that the updated map has high fusion accuracy and strong robustness.
  • FIG. 2A is a schematic diagram of the implementation process of the map update method according to the embodiment of the present application. As shown in FIG. 2A, the method includes the following steps:
  • Step S201 Extract the feature points of each key frame image in the first map to obtain a feature point set.
  • the feature points of each key frame image in the first map are extracted to obtain a feature point set.
  • Step S202 Determine the identification information of each feature point in the feature point set and the 2D position information of each feature point in the key frame image.
  • the descriptor information (identification information) of the feature point is determined, and the 2D position information can be considered as the 2D coordinates of the feature point.
  • Step S203 Map each 2D location information to the coordinate system where the first map is located to obtain the 3D location information of each feature point.
  • the above steps S201 to S203 give a way to implement "extract the key frame image in the first map to obtain the first image feature". In this way, each key frame image in the first map is obtained.
  • Step S204 Determine the ratios of different sample feature points in the feature point set, respectively, to obtain a first ratio vector.
  • the first ratio vector may be pre-stored in the preset word bag model corresponding to the first map.
  • the first ratio vector is obtained from the preset bag-of-words model.
  • the multiple sample feature points are different from each other.
  • the preset bag-of-words model includes a plurality of different sample feature points and the ratio of the plurality of sample feature points to the feature points contained in the key frame image.
  • the first ratio vector may be based on the number of sample images, the number of times the sample feature points appear in the sample image, the number of times the sample feature points appear in the key frame image to be matched, and the sample feature points appearing in the key frame image to be matched. The total is determined; as shown in formula (1):
  • N is the number of sample images (that is, the first number)
  • n i is the number of times the sample feature point w i appears in the sample image (that is, the first number)
  • I t is the image I collected at time t
  • T the number that appears in the key frame image I w i is the sample feature points collected at time (i.e., the number of times a second)
  • the total number of sample feature points as the key frame image I t appeared i.e., a second number).
  • the w-dimensional floating point number vector of each key frame image that is, the ratio vector
  • the ratio vector can also be used as the feature information of the preset bag of words model.
  • Step S205 Obtain a second ratio vector.
  • the second ratio vector is the ratio of the multiple sample feature points among the feature points contained in the key frame image in the second map;
  • the second ratio vector is a pre-stored word
  • the determination process of the second ratio vector is similar to the determination process of the first ratio vector, and both can be determined by using formula (1); and the dimensions of the first ratio vector and the second ratio vector are the same.
  • Step S206 According to the first image feature, the first ratio vector and the second ratio vector, a second image feature is matched from the image features of the key frame image.
  • step S205 can be implemented through the following process:
  • the first step according to the first ratio vector and the second ratio vector, from the image features of the key frame images in the second map, it is determined that the similarity with the first image feature is greater than the second threshold Similar image features.
  • the first ratio vector v 1 of the key frame image to be matched is compared with the second ratio vector v 2 of each key frame image in the second map one by one, and the two ratio vectors are used as shown in formula (2)
  • the similarity between each key frame image in the second map and the key frame image to be matched can be determined, and similar key frame images with similarity greater than or equal to the second threshold can be screened out to obtain a set of similar key frame images.
  • the second step is to determine the similar key frame images to which the similar image features belong to obtain a set of similar key frame images.
  • the third step is to select a second image feature whose similarity to the first image feature meets a preset similarity threshold from the image features of the similar key frame images.
  • the second image feature with the highest similarity to the first image feature for example, first, determine the time difference between the acquisition times of at least two similar key frame images, And the image features of the at least two similar key frame images respectively have a similarity difference with the first image feature; then, the time difference is smaller than a third threshold, and the similarity difference is smaller than a fourth threshold.
  • Frame images are combined to obtain a joint frame image; that is, multiple similar key frame images that are close in acquisition time and close to the similarity of the key frame image to be matched are selected, indicating that these key frame images may be continuous Therefore, the multiple similar key frame images are combined together to form a joint frame image (which can also become an island), so that multiple joint frame images are obtained; finally, from the image characteristics of the joint frame image, select and The second image feature whose similarity of the first image feature meets a preset similarity threshold. For example, firstly, the sum of similarity between the image feature of each key frame image contained in multiple joint frame images and the first image feature is determined separately; in this way, multiple key frame images contained in multiple joint frame images are determined one by one.
  • the joint frame image with the largest sum of similarity is determined as the target joint frame image with the highest similarity to the key frame image to be matched; finally, according to the identification information of the feature points of the target joint frame image and the to-be-matched key frame image Match the identification information of the feature points of the key frame image, and select the second image feature whose similarity with the first image feature meets a preset similarity threshold from the image features of the target joint frame image.
  • the identification information of the feature points of the target joint frame image and the identification information of the feature points of the key frame image to be matched can uniquely identify the feature points of the target joint frame image and the key frame image to be matched, respectively, Therefore, based on these two identification information, the second image feature with the highest similarity to the first image feature can be selected from the image features of the target joint frame image very accurately. Therefore, the accuracy of matching the first image feature of the key frame image to be matched with the second image feature is ensured, and the similarity between the selected second image feature and the first image feature is extremely high.
  • the key frame image containing the second image feature indicates that the key frame image is very similar to the key frame image to be matched, and can be considered as the image most similar to the key frame image to be matched.
  • the above steps S204 to S206 give a way to realize "according to the first image feature, the second image feature is matched from the image feature of the key frame image in the second map," in this mode , By using the preset bag-of-words model to retrieve the second image feature that matches the first image feature from the image features of the key frame image in the second map, ensuring that the second image feature is consistent with the first image feature Similarity.
  • Step S207 Determine the target Euclidean distance between any two feature points contained in the key frame image corresponding to the second image feature and which is less than the first threshold to obtain a target Euclidean distance set.
  • the target Euclidean distance that is less than the first threshold can also be regarded as first determining the minimum Euclidean distance from a plurality of Euclidean distances, and then determining whether the minimum Euclidean distance is less than the first threshold, and if it is less than, then determining the minimum Euclidean distance
  • the distance is the target Euclidean distance, then the target Euclidean distance set is the set with the smallest Euclidean distance among multiple Euclidean distance sets.
  • Step S208 if the number of target Euclidean distances included in the target Euclidean distance set is greater than a fifth threshold, the 3D position information of the feature points of the key frame image corresponding to the second image feature corresponds to the first image feature The 2D position information of the feature points of the key frame image determines the second position information.
  • the number of target Euclidean distances contained in the target Euclidean distance set is greater than the fifth threshold, it means that the number of target Euclidean distances is large enough, and it also means that there are enough feature points that match the features of the first image.
  • the similarity between the key frame image of the image feature and the key frame image to be matched is sufficiently high.
  • the 3D location information of the feature points of the key frame image of the second image feature and the 2D location information of the feature points of the key frame image to be matched corresponding to the first image feature are used as the input of the PnP algorithm.
  • Matching the 3D position information of the feature points in the current frame of the key frame image can solve the position information of the image acquisition device.
  • the above steps S206 to S208 give a way to realize "according to the first image feature and the second image feature, determine the first image acquisition device in the second coordinate system where the second map is located.
  • the 2D and 3D position information of the key frame image is considered at the same time, and the position and posture can be provided in the positioning result at the same time, so the accuracy of determining the position of the image acquisition device is improved, so that The coordinates of the image acquisition device of the key frame image in the first map are effectively converted, and the key frame image in the first map is better merged into the second map, which ensures the fusion accuracy and robustness of the map.
  • Step S209 Acquire first position information of the image acquisition device used to acquire the key frame image to be matched in the first coordinate system where the first map is located.
  • the image acquisition device corresponding to each key frame image in the first map has a piece of first location information.
  • the first position information can be understood as the coordinate value of the image acquisition device in the first coordinate system.
  • Step S210 Add each key frame image in the first map to the second map according to the first location information and the second location information to obtain an updated second map.
  • the key frame image to be matched is obtained through the image acquisition device, the constructed preset map is loaded, and the second image corresponding to the key frame image to be matched is retrieved using the preset bag of words model
  • the key frame image corresponding to the feature, and finally, the 2D location information of the feature point of the key frame image to be matched and the 3D location information of the feature point of the key frame image are used as the input of the PnP algorithm to obtain the current key frame image to be matched
  • the position of the camera in the second map so that the coordinates of the image acquisition device corresponding to the key frame image in the first map are converted into coordinates that match the second coordinate system of the second map, and then the first map
  • the key frame images of the first map can be accurately integrated into the second map; in this way, the key frame images in the first map can be used to determine the coordinates of the image acquisition device corresponding to the key frame images in the second map in the second coordinate system, thereby Combine the two maps more accurately to get the updated second map.
  • FIG. 2B is a schematic diagram of another implementation process of creating a preset map according to an embodiment of the application. As shown in FIG. 2B, the method includes the following steps:
  • Step S221 From the sample image library, select key frame images meeting preset conditions to obtain a set of key frame images.
  • a preset number of corner points are selected from the sample image; the corner points are pixels in the sample image that are significantly different from the preset number of surrounding pixels; for example, 150 corner points are selected point.
  • the second step if the number of identical corner points contained in two sample images with adjacent acquisition times is greater than or equal to the sixth threshold, it is determined that the scene corresponding to the sample image is a continuous scene; the acquisition times of the two sample images are adjacent, It can also be understood as two consecutive sample images to determine the number of the same corner points contained in the two sample images. The larger the number, the higher the correlation between the two sample images and the higher the correlation between the two sample images.
  • the image is an image from a continuous scene. Continuous scenes, such as a single indoor environment, such as a bedroom, a living room, or a single meeting room.
  • the third step is to determine that the scene corresponding to the sample image is a discrete scene if the number of the same corner points contained in the two sample images adjacent to the acquisition time is less than the sixth threshold.
  • the scene corresponding to the sample image is a discrete scene
  • the scene corresponding to the sample image is a continuous scene
  • Step S222 Extract the image features of each key frame image to obtain a key image feature set.
  • the image feature of the key frame includes: 2D location information, 3D location information of the feature point of the key frame image, and identification information that can uniquely identify the feature point.
  • Step S223 Determine the ratio of each sample feature point in the key frame image to obtain a ratio vector set.
  • the different sample feature points and the ratio vector set are stored in the preset word bag model, so that the preset word bag model is used to retrieve the key frame image to be matched from the key frame image
  • the key frame image corresponding to the second image feature can be implemented through the following process:
  • the first average number of times is determined according to the first number of sample images contained in the sample image library and the first number of times that the i-th sample feature point appears in the sample image library.
  • i is an integer greater than or equal to 1; the first average number is used to indicate the average number of times the i-th sample feature point appears in each sample image; for example, the first number of the sample image is N, and the i-th sample feature
  • the first number of times a point appears in the sample image library is n i
  • the first average number of times idf(i) can be obtained by formula (1).
  • the second average number tf(i,I t ) can be obtained by formula (1).
  • the ratio of the sample feature point in the key frame image is obtained, and the ratio vector set is obtained. For example, according to formula (1), multiply the first average number by the second average number to obtain the ratio vector
  • Step S224 Store the ratio vector set and the key image feature set to obtain the first map.
  • the ratio vector set corresponding to the key frame image and the key image feature set are stored in the first map, so that when the image acquisition device is located, the ratio vector set and the preset word bag model determined to be matched
  • the ratio vector sets corresponding to the key frame images are compared to determine the key frame image corresponding to the second image feature that is highly similar to the key frame image to be matched from the key image feature set.
  • the above steps S221 to S224 give the process of constructing local maps, such as the first map and the second map. Repeating the above steps S221 to S224, the second map can be obtained, and the map corresponding to any scene can also be obtained.
  • FIG. 2C is a schematic diagram of another implementation process of the map update method according to the embodiment of the application. As shown in FIG. 2C, the method includes the following steps:
  • Step S231 From the sample image library, select key frame images meeting preset conditions to obtain a set of key frame images.
  • Step S232 Extract the image features of each key frame image to obtain a key image feature set.
  • Step S233 Extract the feature points of the sample image to obtain a sample feature point set containing different feature points.
  • Step S234 Determine the ratio of each sample feature point in the key frame image to obtain a ratio vector set.
  • Step S235 Store the ratio vector set and the key image feature set to obtain the first map and the second map.
  • the above steps S231 to S235 complete the creation process of the preset map, and store the image features and ratio vector sets of the key frame images in the preset map, so that the key frame images in the second map can be collected according to the ratio vector set.
  • the second image feature matching the image feature of the key frame image to be matched is searched out of the image features.
  • Step S236 Extract the first image feature of the key frame image to be matched from the image feature of the key frame image to be matched in the first map.
  • the first map and the second map need to be loaded first.
  • Step S237 According to the first image feature, the second image feature is matched from the image feature of the key frame image in the second map.
  • Step S238 According to the first image feature and the second image feature, determine the second position information of the image acquisition device in the second coordinate system where the second map is located.
  • the above steps S236 to S238 give the process of realizing the positioning of the image acquisition device.
  • the second image whose features are highly similar to the first image is matched from the key frame images stored in the preset map.
  • the image features and then, using the 2D position information and the 3D position information of the two image features, the second position information of the collection device in the second coordinate system can be finally determined.
  • Step S239 Acquire first position information of the image acquisition device used to acquire the key frame image to be matched in the first coordinate system where the first map is located.
  • Step S240 Add each key frame image in the first map to the second map according to the first location information and the second location information to obtain an updated second map.
  • the 2D and 3D position information of the key frame image is used at the same time to ensure the accuracy of the positioning result of the acquisition device, the positioning success rate is high, and the robustness is strong.
  • the key in the first map The frame images are fused into the second map to ensure the fusion accuracy of the fused map.
  • FIG. 3 is a schematic diagram of another implementation process of the map update method according to the embodiment of the application. As shown in FIG. 3, the method includes the following steps:
  • Step S301 According to the scene to which the sample image belongs, different key frame image selection methods are selected.
  • the scene to which the sample image belongs includes: discrete scene or continuous scene;
  • the key frame image selection methods include manual selection and automatic selection.
  • Manual selection requires the creator to manually select the key frame images that need to be included in the map, while automatic selection is The method of automatically selecting an image as a key frame image according to the frame rate or parallax.
  • 150 FAST feature corner points are extracted from each key frame image.
  • the ratio of the same corner points in two consecutive key frame images is defined as the corner tracking rate.
  • the embodiment of the present application defines a scene with an ordered key frame image sequence and an average corner tracking rate greater than 30% as a continuous scene, otherwise it is a discrete scene.
  • the key frame image selection method of continuous scenes uses automatic selection method; the key frame image selection method of discrete scenes uses manual selection method.
  • the continuous scene is suitable for a single indoor environment, such as a bedroom, a living room, a single meeting room, etc.; a discrete scene is more suitable for use in multiple indoor environments, such as multiple rooms in a building, or multiple meetings on a first floor Room etc.
  • the key frame image selection strategies of continuous scenes and discrete scenes are different, and the applicable scenes are different. In this way, for indoor discrete or continuous scenes, different key frame image selection methods are used to extract image features for map construction, so that the positioning process does not depend on external base station equipment, low cost, high positioning accuracy, and strong robustness.
  • Step S302 Use the camera to collect key frame images.
  • the camera may be a monocular camera or a binocular camera.
  • step S303 the image features in the key frame image are extracted in real time during the acquisition process.
  • image feature extraction is a process of interpretation and annotation of key frame images.
  • step S303 it is necessary to extract the 2D position information, 3D position information, and identification information of the feature point of the key frame image (ie the descriptor information of the feature point); among them, the 3D position information of the feature point of the key frame image is the key
  • the 2D position information of the feature points of the frame image is mapped in the three-dimensional coordinate system where the preset map is located.
  • the number of extraction is 150 (150 is the empirical value, the number of feature points is too small, the tracking failure rate is high, the number of feature points is too much, which affects the efficiency of the algorithm), which is used for images Tracking; and extract the descriptor of the feature point for feature point matching; secondly, the 3D position information (ie depth information) of the feature point is calculated by the triangulation method to determine the location of the acquisition camera.
  • Step S304 Determine the ratio of each sample feature point in the key frame image in real time during the acquisition process to obtain a ratio vector.
  • step S304 can be understood as, during the acquisition of the key frame image, for the current frame image, the ratio vector of the key frame image is extracted in real time.
  • the word bag model is described in the form of a vocabulary tree.
  • the bag-of-words model includes sample image database 41, which is the root node of the vocabulary tree; sample images 42, 43, and 44, namely leaf nodes 42, 43; sample feature points 1 to 3 are different sample feature points in sample image 42 , The present feature points 4 to 6 are different sample feature points in the sample image 43, and the present feature points 7 to 9 are different sample feature points in the sample image 44.
  • w is the number of feature points extracted from the sample image of the bag of words model. So there are a total of w sample feature points in the bag of words model.
  • Each sample feature point will score the key frame image, and the score value is a floating point number from 0 to 1, so that each key frame image can be represented by a w-dimensional floating point number.
  • This w-dimensional vector is the output of the bag of words model Ratio vector The scoring process is shown in formula (1).
  • the above steps S301 to S304 construct an offline local map that depends on the key frame image, namely the first map and the second map.
  • the first map and the second map store the image characteristics of the key frame image in a binary format ( Including: 2D location information, 2D location information and identification information, such as 2D coordinates, 3D coordinates, and descriptor information) to the local device, when the image acquisition device needs to be processed, the first map and the second map will be loaded use.
  • Step S305 Load the constructed first map and second map.
  • the second coordinate system of the second map is taken as the global coordinate system.
  • Step S306 Extract the image feature of the key frame image to be matched in the first map to obtain the first image feature.
  • the key frame image to be matched is any one of the key frame images in the first map.
  • Step S307 Search for the key frame image corresponding to the second image feature of the current frame of the key frame image to be matched in the second map through the bag of words model.
  • the retrieval of the key frame image corresponding to the second image feature of the current frame of the key frame image to be matched in the second map through the bag-of-words model can be understood as using the feature information of the bag-of-words model, that is, the ratio vector set for retrieval The key frame image corresponding to the second image feature of the current frame of the key frame image to be matched in the second map.
  • the step S307 can be implemented through the following process:
  • the first step is to find the similarity between the current frame of the key frame image to be matched and each key frame image in the second map.
  • the calculation method of the similarity s(v 1 , v 2 ) is shown in formula (2).
  • v 1 and v 2 respectively represent the first ratio vector of each sample feature point contained in the bag of words model in the current frame of the key frame image to be matched, and each sample feature point in the second map The second ratio vector of the key frame image in. If the bag of words model contains w sample feature points, then the first ratio vector and the second ratio vector are both w-dimensional vectors. The similar key frame images whose similarity reaches the second threshold among the key frame images in the second map are filtered out to become a set of similar key frame images.
  • the second step is to select similar key frame images whose time stamp difference is less than the third threshold and similarity difference less than the fourth threshold from the set of similar key frame images to obtain joint frame images (or called islands).
  • the second step can be understood as selecting similar key frame images with close timestamps and similarity matching scores in the set of similar key frame images to be combined together to form islands; in this way, the set of similar key frame images is divided into Multiple joint frame images (ie multiple islands) are created.
  • the ratio of the similarity between the first key frame image and the last key frame image in the joint frame image is very small, and the similarity ratio As shown in formula (3):
  • the third step is to respectively determine the sum of similarity between the image features of each key frame image contained in the multiple joint frame images and the first image feature, as shown in formula (4),
  • the fourth step is to determine the joint frame image with the largest sum of similarity as the target joint frame image with the highest similarity to the key frame image to be matched, and find out the key frame to be matched from the target joint frame image The key frame image corresponding to the second image feature with the highest similarity in the current frame of the image.
  • Step S308 Using the PnP algorithm, determine the location information of the second location information in the map coordinate system of the image acquisition device corresponding to the key frame image to be matched in the second coordinate system where the second map is located.
  • step S308 can be implemented through the following steps:
  • the Nth feature point F CN of the current frame X C of the key frame image to be matched traverse all the feature points of the key frame image X 3 corresponding to the second image feature, and determine the key frame corresponding to the second image feature The Euclidean distance between any two feature points in the image.
  • the current frame Xc51 of the key frame image to be matched and the key frame image X352 corresponding to the second image feature matching the current frame Xc51.
  • the second step is to select the group with the smallest Euclidean distance (ie the target Euclidean distance set) for threshold judgment. If it is less than the first threshold and determine the target Euclidean distance, the target Euclidean distance set will be formed; otherwise, the target Euclidean distance set will not be formed and skip Go to the first step until all the feature points of X C are traversed, and then enter the third step. For example, as shown in Figure 5A, by comparing multiple Euclidean distances, a set of minimum Euclidean distance combinations ⁇ F 1 , F 2 , F 3 ⁇ are obtained.
  • the third step is to form the target Euclidean distance set, which can be expressed as ⁇ F 1 , F 2 , F 3 ⁇ . If the number of elements in the target Euclidean distance set is greater than the fifth threshold, proceed to the fourth step, otherwise the algorithm ends and the matching frame is output X 3 location information.
  • the input of the PnP algorithm is the 3D coordinates of the feature points in the key frame image containing the second image feature and the 2D coordinates of the feature points in the current frame of the key frame image to be matched.
  • the output of the algorithm is the current frame of the key frame image to be matched. The position of the corresponding image acquisition device in the second coordinate system of the second map.
  • the PnP algorithm does not directly obtain the camera pose matrix according to the matching pair sequence, but first obtains the 2D coordinates of the feature points in the current frame of the key frame image to be matched. The feature in the current frame of the key frame image to be matched in the current coordinate system The 3D coordinates of the points are then calculated based on the 3D coordinate system in the map coordinate system and the 3D coordinates of the feature points in the current frame of the key frame image to be matched in the current coordinate system to solve the camera pose.
  • the solution of the PnP algorithm starts from the law of cosines. Suppose the center of the current coordinate system is point O, and A, B and C are three characteristic points in the current frame of the key frame image to be matched, as shown in Figure 5B:
  • the location of the collection device is determined through the transformation from the map coordinate system to the current coordinate system.
  • the above steps S305 to S308, load the constructed offline map for the key frame image to be matched collected by the image acquisition device, and retrieve the second image of the key frame image to be matched from the key frame images in the second map through the bag of words model The key frame image corresponding to the feature, and finally the PnP algorithm is used to solve the precise pose of the current camera in the second map to determine the position and posture of the device in the second coordinate system, so that the image in the first map can be more accurately
  • the key frame image is added to the second map to achieve map update.
  • Step S309 Adjust the coordinates in the first coordinate system of the image acquisition device corresponding to each key frame image in the first map, so that the adjusted coordinates match the second coordinate system, and the coordinate adjusted image
  • the key frame image corresponding to the acquisition device is added to the second map to obtain an updated second map.
  • firstly determine the first position information of the image acquisition device in the first coordinate system; secondly, determine the second position information of the image acquisition device in the second coordinate system; again, based on the first position information and the second Position information, determine the rotation matrix and translation vector of the first coordinate system relative to the second coordinate system; use the rotation matrix and translation vector to adjust the coordinates of the image acquisition device in the first coordinate system, so that the adjusted image acquisition The coordinates of the device match the second coordinate system.
  • steps S305 to S309 are repeatedly executed to merge other local maps into the second coordinate system (ie, the global coordinate system) to complete the fusion of the global map.
  • the 2D coordinates and 3D coordinates of the key frame image are considered at the same time, and the 3D coordinates of the acquisition device can be provided in the positioning result, which improves the positioning accuracy of the image acquisition device and ensures that the key in the first map
  • the fusion accuracy of the frame image fusion into the second map makes the updated map robust.
  • An embodiment of the present application provides a map update device, which includes each module included and each unit included in each module, which can be implemented by a processor in a computer device; of course, it can also be implemented by a specific logic circuit;
  • the processor may be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA).
  • FIG. 6 is a schematic diagram of the composition structure of a map updating apparatus according to an embodiment of the application.
  • the apparatus 600 includes: a first obtaining module 601, a first matching module 602, and a first updating module 603, wherein: An acquiring module 601 is configured to acquire a first image feature in a first map; the first matching module 602 is configured to match a second image feature from a second map according to the first image feature; An update module 603 is configured to add each key frame image in the first map to the second map according to the first image feature and the second image feature to obtain an updated second map.
  • the first acquisition module 601 includes: a first extraction sub-module configured to extract the key frame image to be matched in the first map to obtain the first image feature; wherein, the key to be matched
  • the frame image is a key frame image in the first map.
  • the first matching module 602 includes: a first matching sub-module configured to retrieve the key frames from the second map according to the first image feature Among the image features of the frame image, the second image feature is matched.
  • the first update module 603 includes: a first acquisition sub-module configured to acquire the first coordinates of the image acquisition device used to acquire the key frame image to be matched on the first map. First location information in the system; a first determining sub-module configured to determine the second coordinate system where the image acquisition device is located on the second map based on the first image feature and the second image feature The second location information in the; a first update submodule configured to add each key frame image in the first map to the second map according to the first location information and the second location information , Get the updated second map.
  • the first update submodule includes: a first determining unit configured to determine the first coordinate system and the second coordinate according to the first position information and the second position information
  • the first adjustment unit is configured to adjust the coordinates in the first coordinate system of the image acquisition device corresponding to each key frame image in the first map according to the conversion relationship, so that The adjusted coordinates of the image capture device corresponding to each key frame image match the second coordinate system
  • the first adding unit is configured to adjust the key frame image corresponding to the image capture device after each coordinate adjustment, Add to the second map to obtain the updated second map.
  • the scene information corresponding to the first map and the second map are at least partially the same.
  • the first image feature includes: 2D location information, 3D location information, and identification information of the feature point of the key frame image in the first map
  • the second image feature includes: the second The 2D position information, 3D position information, and identification information of the feature points of the key frame image in the map
  • the 3D position information is obtained by mapping the 2D position information in the coordinate system where the 2D position information is located .
  • the first extraction sub-module includes: a first extraction unit configured to extract feature points of each key frame image in the first map to obtain a feature point set; and a second determination unit configured to To determine the identification information of each feature point in the feature point set and the 2D location information of each feature point in the key frame image; the first mapping unit is configured to map each 2D location information to In the coordinate system where the first map is located, 3D position information of each of the feature points is obtained.
  • the first matching submodule includes: a third determining unit configured to determine the ratios of different sample feature points in the feature point set to obtain the first ratio vector; A unit configured to obtain a second ratio vector, where the second ratio vector is the ratio of the multiple sample feature points among the feature points included in the key frame image in the second map; a first matching unit And configured to match the second image feature from the image features of the key frame image in the second map according to the first image feature, the first ratio vector and the second ratio vector.
  • the first matching unit includes: a first determining subunit configured to obtain an image from a key frame image in the second map according to the first ratio vector and the second ratio vector Among the features, similar image features whose similarity with the first image feature is greater than a second threshold is determined; the second determining subunit is configured to determine the similar key frame images to which the similar image features belong to obtain a set of similar key frame images The first selection subunit is configured to select a second image feature whose similarity with the first image feature meets a preset similarity threshold from the image features of the similar key frame images.
  • the first selection subunit is configured to: determine the time difference between the acquisition times of at least two similar key frame images, and the image characteristics of the at least two similar key frame images are respectively compared with all The similarity difference of the first image feature; the similar key frame images whose time difference is smaller than the third threshold and the similarity difference is smaller than the fourth threshold are combined to obtain a joint frame image; from the image of the joint frame image Among the features, a second image feature whose similarity with the first image feature meets a preset similarity threshold is selected.
  • the first selection subunit is configured to: respectively determine the sum of similarity between the image feature of each key frame image contained in the multiple joint frame images and the first image feature;
  • the joint frame image with the largest sum is determined as the target joint frame image with the highest similarity to the key frame image to be matched; according to the identification information of the characteristic points of the target joint frame image and the characteristic points of the key frame image to be matched Identification information, selecting a second image feature whose similarity with the first image feature meets a preset similarity threshold from the image features of the target joint frame image.
  • the device further includes: a first determining module configured to determine a target Euclidean distance between any two feature points contained in the key frame image corresponding to the second image feature that is less than a first threshold, Obtain a target Euclidean distance set; correspondingly, the first determining submodule includes: a fourth determining unit configured to, if the number of target Euclidean distances included in the target Euclidean distance set is greater than a fifth threshold, based on the first The 3D position information of the feature point of the key frame image corresponding to the second image feature and the 2D position information of the feature point of the key frame image corresponding to the first image feature determine the second position information.
  • the device further includes: a first selection module configured to select key frame images satisfying preset conditions from a sample image library to obtain a set of key frame images; and a first extraction module configured to extract each The image features of the key frame image are used to obtain the key image feature set; the second extraction module is configured to extract the feature points of the sample image to obtain a sample feature point set containing different feature points; the second determination module is configured to determine each sample The feature point is the ratio of the key frame image to obtain a ratio vector set; the first storage module is configured to store the ratio vector set and the key image feature set to obtain the first map.
  • a first selection module configured to select key frame images satisfying preset conditions from a sample image library to obtain a set of key frame images
  • a first extraction module configured to extract each The image features of the key frame image are used to obtain the key image feature set
  • the second extraction module is configured to extract the feature points of the sample image to obtain a sample feature point set containing different feature points
  • the second determination module is configured to determine each sample The feature point is the ratio
  • the device further includes: a second selection module configured to select a preset number of corner points from the sample image; a third determination module configured to select two sample images with adjacent acquisition times The number of the same corner points included is greater than or equal to the sixth threshold, and it is determined that the scene corresponding to the sample image is a continuous scene; the fourth determining module is configured to if the same number of corner points are included in two sample images with adjacent acquisition times If it is less than the sixth threshold, it is determined that the scene corresponding to the sample image is a discrete scene.
  • the first selection module includes: a first selection sub-module configured to select a key frame image from a sample image library according to an input selection instruction if the scene corresponding to the sample image is a discrete scene;
  • the second selection sub-module is configured to select a key frame image from the sample image library according to a preset frame rate or disparity if the scene corresponding to the sample image is a continuous scene.
  • the second determining module includes: a second determining sub-module configured to determine according to the first number of sample images contained in the sample image library and the i-th sample feature point in the sample image library. A number of times, determining the first average number; where i is an integer greater than or equal to 1; the first average number is configured to indicate the average number of times the i-th sample feature point appears in each sample image; third The determining submodule is configured to determine the second number of sample feature points contained in the j-th key frame image and the second number of times the i-th sample feature point appears in the j-th key frame image.
  • the sub-module is configured to obtain the ratio of the sample feature points in the key frame image according to the first average number and the second average number, and obtain the ratio vector set.
  • the above map update method is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.
  • the technical solutions of the embodiments of the present application can be embodied in the form of software products in essence or the parts that contribute to related technologies.
  • the computer software products are stored in a storage medium and include several instructions to enable The automatic test line of the device including the storage medium executes all or part of the method described in each embodiment of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read only memory (Read Only Memory, ROM), magnetic disk or optical disk and other media that can store program codes.
  • an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps in the map update method provided in the above embodiments are implemented.
  • the description of the foregoing storage medium and device embodiments is similar to the foregoing description of the method embodiment, and has similar beneficial effects as the method embodiment.
  • the description of the method embodiments of this application please refer to the description of the method embodiments of this application for understanding.
  • “one embodiment” or “an embodiment” mentioned throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the present application.
  • the disclosed device and method may be implemented in other ways.
  • the device embodiments described above are merely illustrative.
  • the division of the units is only a logical function division, and there may be other divisions in actual implementation, such as: multiple units or components can be combined, or It can be integrated into another system, or some features can be ignored or not implemented.
  • the coupling, or direct coupling, or communication connection between the components shown or discussed may be indirect coupling or communication connection through some interfaces, devices or units, and may be electrical, mechanical or other forms of.
  • the units described above as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments of the present application.
  • the functional units in the embodiments of the present application can all be integrated into one processing unit, or each unit can be individually used as a unit, or two or more units can be integrated into one unit;
  • the unit can be implemented in the form of hardware, or in the form of hardware plus software functional units.
  • the foregoing program can be stored in a computer readable storage medium.
  • the execution includes The steps of the foregoing method embodiment; and the foregoing storage medium includes: various media that can store program codes, such as a mobile storage device, a read only memory (Read Only Memory, ROM), a magnetic disk, or an optical disk.
  • ROM Read Only Memory
  • the above-mentioned integrated unit of this application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer readable storage medium.
  • the technical solutions of the embodiments of the present application can be embodied in the form of software products in essence or the parts that contribute to related technologies.
  • the computer software products are stored in a storage medium and include several instructions to enable The equipment automatic test line executes all or part of the method described in each embodiment of the present application.
  • the aforementioned storage media include: removable storage devices, ROMs, magnetic disks or optical discs and other media that can store program codes.
  • each key frame image in the first map is added to the second map to obtain an updated second map; in this way, by extracting the key frame image in the first map of the local map Image feature, and then match the image feature with the image feature of the key frame image in another map to obtain the key frame image corresponding to the second image feature, so that based on the matching image features in different maps, multiple The map is merged, thereby realizing the update of the map and improving the accuracy of the updated map.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Remote Sensing (AREA)
  • Databases & Information Systems (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Automation & Control Theory (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Library & Information Science (AREA)
  • Image Analysis (AREA)
  • Processing Or Creating Images (AREA)

Abstract

一种地图更新方法,包括:获取第一地图中的第一图像特征;根据所述第一图像特征,从第二地图匹配出第二图像特征;其中,所述第一地图和所述第二地图对应的场景信息部分相同;根据所述第一图像特征和所述第二图像特征,将所述第一地图中每一关键帧图像添加到所述第二地图中,得到更新的第二地图。还同时提供了一种地图更新装置、终端及存储介质。

Description

地图更新方法及装置、终端、存储介质
相关申请的交叉引用
本申请实施例基于申请号为201910578745.9、申请日为2019年06月28日的中国专利申请提出,并要求该中国专利申请的优先权,该中国专利申请的全部内容在此以引入方式并入本申请实施例。
技术领域
本申请涉及室内定位技术,涉及但不限于地图更新方法及装置、终端、存储介质。
背景技术
在相关技术中,通过两局部地图的直方图相关性构建全局地图,但是计算直方图的角度偏移量和平移偏移量是依赖于点云的法相特征;这样,由于点云的法相特征的准确性不高,易出现误差,会导致所得到的地图的精确性较低。
发明内容
有鉴于此,本申请实施例为解决相关技术中存在的至少一个问题而提供一种地图更新方法及装置、终端、存储介质。
本申请实施例的技术方案是这样实现的:
本申请实施例提供了一种地图更新方法,所述方法包括:
获取第一地图中的第一图像特征;
根据所述第一图像特征,从第二地图匹配出第二图像特征;其中,所述第一地图和所述第二地图对应的场景信息部分相同;
根据所述第一图像特征和所述第二图像特征,将所述第一地图中每一关键帧图像添加到所述第二地图中,得到更新的第二地图。
在上述方法中,所述获取第一地图中的第一图像特征,包括:
提取所述第一地图中的待匹配关键帧图像,得到第一图像特征;其中,所述待匹配关键帧图像为所述第一地图中的关键帧图像;
对应地,所述根据所述第一图像特征,从第二地图匹配出第二图像特征,包括:
根据所述第一图像特征,从第二地图中的关键帧图像的图像特征中,匹配出所述第二图像特征。
在上述方法中,根据所述第一图像特征和所述第二图像特征,将所述第一地图中每一关键帧图像添加到所述第二地图中,得到更新的第二地图,包括:
获取用于采集所述待匹配关键帧图像的图像采集设备在所述第一地图所处的第一坐标系中的第一位置信息;
根据所述第一图像特征和所述第二图像特征,确定所述图像采集设备在所述第二地图所处的第二坐标系中的第二位置信息;
根据所述第一位置信息和所述第二位置信息,将所述第一地图中每一关键帧图像添加到所述第二地图中,得到更新的第二地图。
在上述方法中,所述根据所述第一位置信息和所述第二位置信息,将所述第一地图中每一关键帧图像添加到所述第二地图中,得到更新的第二地图,包括:
根据所述第一位置信息和所述第二位置信息,确定所述第一坐标系和所述第二坐标系之间的转换关系;
根据所述转换关系,对所述第一地图中的每一关键帧图像对应的图像采集设备在第一坐标系中 的坐标进行调整,使得所述每一关键帧图像对应的图像采集设备的调整后的坐标与所述第二坐标系相匹配;
将每一坐标调整后的图像采集设备对应的关键帧图像,添加到所述第二地图中,得到所述更新的第二地图。
在上述方法中,所述第一地图和所述第二地图对应的场景信息至少部分相同。
在上述方法中,所述第一图像特征包括:所述第一地图中的关键帧图像的特征点的二维(2-Dimensional,2D)位置信息、(3-Dimensional,3D)位置信息和标识信息;
所述第二图像特征包括:所述第二地图中的关键帧图像的特征点的二维位置信息、三维位置信息和标识信息;其中,所述三维位置信息是将所述二维位置信息映射在所述二维位置信息所处的坐标系中得到的。
在上述方法中,所述对第一地图中的关键帧图像进行提取,得到第一图像特征,包括:
提取所述第一地图中的每一关键帧图像的特征点,得到特征点集合;
确定所述特征点集合中每一特征点的标识信息和每一所述特征点在所述关键帧图像中的二维位置信息;
将每一二维位置信息分别映射在所述第一地图所处的坐标系中,得到每一所述特征点的三维位置信息。
在上述方法中,所述根据所述第一图像特征,从第二地图中的关键帧图像的图像特征中,匹配出第二图像特征,包括:
分别确定不同的样本特征点在所述特征点集合中所占的比值,得到第一比值向量;
获取第二比值向量,所述第二比值向量为所述多个样本特征点在所述第二地图中的关键帧图像中包含的特征点中所占的比值;
根据所述第一图像特征、所述第一比值向量和所述第二比值向量,从所述第二地图中的关键帧图像的图像特征中,匹配出第二图像特征。
在上述方法中,所述根据所述第一图像特征、所述第一比值向量和所述第二比值向量,从所述第二地图中的关键帧图像的图像特征中,匹配出第二图像特征,包括:
根据所述第一比值向量和所述第二比值向量,从所述第二地图中的关键帧图像的图像特征中,确定与所述第一图像特征的相似度大于第二阈值的相似图像特征;
确定所述相似图像特征所属的相似关键帧图像,得到相似关键帧图像集合;
从所述相似关键帧图像的图像特征中,选择与所述第一图像特征相似度满足预设相似度阈值的第二图像特征。
在上述方法中,所述从所述相似关键帧图像的图像特征中,选择与所述第一图像特征相似度满足预设相似度阈值的第二图像特征,包括:
确定至少两个所述相似关键帧图像的采集时间之间的时间差,和所述至少两个相似关键帧图像的图像特征分别与所述第一图像特征的相似度差;
将所述时间差小于第三阈值,且所述相似度差小于第四阈值的相似关键帧图像进行联合,得到联合帧图像;
从所述联合帧图像的图像特征中,选择与所述第一图像特征相似度满足预设相似度阈值的第二图像特征。
在上述方法中,所述从所述联合帧图像的图像特征中,选择与所述第一图像特征相似度满足预设相似度阈值的第二图像特征,包括:
分别确定多个联合帧图像中包含的每一关键帧图像的图像特征与所述第一图像特征的相似度之和;
将相似度之和最大的联合帧图像,确定为与所待匹配关键帧图像的相似度最高的目标联合帧图像;
根据目标联合帧图像的特征点的标识信息和所述待匹配关键帧图像的特征点的标识信息,从所述目标联合帧图像的图像特征中,选择与所述第一图像特征相似度满足预设相似度阈值的第二图像特征。
在上述方法中,在所述根据所述第一图像特征和所述第二图像特征,确定所述图像采集设备在所述第二地图所处的第二坐标系中的第二位置信息之前,所述方法还包括:
确定所述第二图像特征对应的关键帧图像中包含的任意两个特征点之间,小于第一阈值的目标欧式距离,得到目标欧式距离集合;
对应地,所述根据所述第一图像特征和所述第二图像特征,确定所述图像采集设备在所述第二地图所处的第二坐标系中的第二位置信息,包括:
如果所述目标欧式距离集合中包含的目标欧式距离的数量大于第五阈值,基于所述第二图像特征对应的关键帧图像的特征点的三维位置信息和所述第一图像特征对应的关键帧图像的特征点的二维位置信息,确定所述第二位置信息。
在上述方法中,在所述对第一地图中的关键帧图像进行提取,得到第一图像特征之前,所述方法还包括:
从样本图像库中,选择满足预设条件的关键帧图像,得到关键帧图像集合;
提取每一关键帧图像的图像特征,得到关键图像特征集合;
提取样本图像的特征点,得到包含不同的特征点的样本特征点集合;
确定每一样本特征点,在所述关键帧图像的中所占的比值,得到比值向量集合;
存储所述比值向量集合和所述关键图像特征集合,得到所述第一地图。
在上述方法中,在所述从样本图像库中,选择满足预设条件的关键帧图像,得到关键帧图像集合之前,所述方法还包括:
从所述样本图像中选择预设数量的角点;
如果采集时间相邻的两个样本图像中包含的相同的角点数量大于等于第六阈值,确定所述样本图像对应的场景为连续场景;
如果采集时间相邻的两个样本图像中包含的相同的角点数量小于第六阈值,确定所述样本图像对应的场景为离散场景。
在上述方法中,从样本图像库中,选择满足预设条件的关键帧图像,得到关键帧图像集合,包括:
如果所述样本图像对应的场景为离散场景,根据输入的选择指令,从样本图像库中选择关键帧图像;
如果所述样本图像对应的场景为连续场景,根据预设的帧率或视差,从样本图像库中选择关键帧图像。
在上述方法中,所述确定每一样本特征点,在所述关键帧图像的中所占的比值,得到比值向量集合,包括:
根据样本图像库中包含的样本图像的第一数量和第i个样本特征点在样本图像库中出现的第一次数,确定第一平均次数;其中,i为大于等于1的整数;所述第一平均次数用于表明所述第i个样本特征点平均在每一样本图像中出现的次数;
根据所述第i个样本特征点在第j个关键帧图像中出现的第二次数和所述第j个关键帧图像中包含的样本特征点的第二数量,确定第二平均次数;其中,j为大于等于1的整数;所述第二平均次数用于表明所述第i个样本特征点占据第j个关键帧图像中包含的样本特征点的比例;
根据所述第一平均次数和所述第二平均次数,得到样本特征点在所述关键帧图像的中所占的比值,得到所述比值向量集合。
本申请实施例提供一种地图更新装置,所述装置包括:第一获取模块、第一匹配模块和第一更新模块,其中:
所述第一获取模块,配置为获取第一地图中的第一图像特征;
所述第一匹配模块,配置为根据所述第一图像特征,从第二地图匹配出第二图像特征;其中,所述第一地图和所述第二地图对应的场景信息部分相同;
所述第一更新模块,配置为根据所述第一图像特征和所述第二图像特征,将所述第一地图中每一关键帧图像添加到所述第二地图中,得到更新的第二地图。
在上述装置中,所述第一获取模块,包括:
第一提取子模块,配置为提取所述第一地图中的待匹配关键帧图像,得到第一图像特征;其中,所述待匹配关键帧图像为所述第一地图中的关键帧图像;
对应地,所述第一匹配模块,包括:
第一匹配子模块,配置为根据所述第一图像特征,从第二地图中的关键帧图像的图像特征中,匹配出所述第二图像特征。
在上述装置中,所述第一更新模块,包括:
第一获取子模块,配置为获取用于采集所述待匹配关键帧图像的图像采集设备在所述第一地图所处的第一坐标系中的第一位置信息;
第一确定子模块,配置为根据所述第一图像特征和所述第二图像特征,确定所述图像采集设备在所述第二地图所处的第二坐标系中的第二位置信息;
第一更新子模块,配置为根据所述第一位置信息和所述第二位置信息,将所述第一地图中每一关键帧图像添加到所述第二地图中,得到更新的第二地图。
在上述装置中,所述第一更新子模块,包括:
第一确定单元,配置为根据所述第一位置信息和所述第二位置信息,确定所述第一坐标系和所述第二坐标系之间的转换关系;
第一调整单元,配置为根据所述转换关系,对所述第一地图中的每一关键帧图像对应的图像采集设备在第一坐标系中的坐标进行调整,使得所述每一关键帧图像对应的图像采集设备的调整后的坐标与所述第二坐标系相匹配;
第一添加单元,配置为将每一坐标调整后的图像采集设备对应的关键帧图像,添加到所述第二地图中,得到所述更新的第二地图。
在上述装置中,所述第一地图和所述第二地图对应的场景信息至少部分相同。
在上述装置中,所述第一图像特征包括:所述第一地图中的关键帧图像的特征点的二维位置信息、三维位置信息和标识信息;
所述第二图像特征包括:所述第二地图中的关键帧图像的特征点的二维位置信息、三维位置信息和标识信息;其中,所述三维位置信息是将所述二维位置信息映射在所述二维位置信息所处的坐标系中得到的。
在上述装置中,所述第一提取子模块,包括:
第一提取单元,配置为提取所述第一地图中的每一关键帧图像的特征点,得到特征点集合;
第二确定单元,配置为确定所述特征点集合中每一特征点的标识信息和每一所述特征点在所述关键帧图像中的二维位置信息;
第一映射单元,配置为将每一二维位置信息分别映射在所述第一地图所处的坐标系中,得到每一所述特征点的三维位置信息。
在上述装置中,所述第一匹配子模块,包括:
第三确定单元,配置为分别确定不同的样本特征点在所述特征点集合中所占的比值,得到第一比值向量;
第一获取单元,配置为获取第二比值向量,所述第二比值向量为所述多个样本特征点在所述第二地图中的关键帧图像中包含的特征点中所占的比值;
第一匹配单元,配置为根据所述第一图像特征、所述第一比值向量和所述第二比值向量,从所述第二地图中的关键帧图像的图像特征中,匹配出第二图像特征。
在上述装置中,所述第一匹配单元,包括:
第一确定子单元,配置为根据所述第一比值向量和所述第二比值向量,从所述第二地图中的关键帧图像的图像特征中,确定与所述第一图像特征的相似度大于第二阈值的相似图像特征;
第二确定子单元,配置为确定所述相似图像特征所属的相似关键帧图像,得到相似关键帧图像集合;
第一选择子单元,配置为从所述相似关键帧图像的图像特征中,选择与所述第一图像特征相似度满足预设相似度阈值的第二图像特征。
在上述装置中,所述第一选择子单元,配置为:确定至少两个所述相似关键帧图像的采集时间之间的时间差,和所述至少两个相似关键帧图像的图像特征分别与所述第一图像特征的相似度差;将所述时间差小于第三阈值,且所述相似度差小于第四阈值的相似关键帧图像进行联合,得到联合帧图像;从所述联合帧图像的图像特征中,选择与所述第一图像特征相似度满足预设相似度阈值的第二图像特征。
在上述装置中,所述第一选择子单元,配置为:分别确定多个联合帧图像中包含的每一关键帧图像的图像特征与所述第一图像特征的相似度之和;将相似度之和最大的联合帧图像,确定为与所待匹配关键帧图像的相似度最高的目标联合帧图像;根据目标联合帧图像的特征点的标识信息和所述待匹配关键帧图像的特征点的标识信息,从所述目标联合帧图像的图像特征中,选择与所述第一图像特征相似度满足预设相似度阈值的第二图像特征。
在上述装置中,所述装置还包括:
第一确定模块,配置为确定所述第二图像特征对应的关键帧图像中包含的任意两个特征点之间,小于第一阈值的目标欧式距离,得到目标欧式距离集合;
对应地,所述第一确定子模块,包括:
第四确定单元,配置为如果所述目标欧式距离集合中包含的目标欧式距离的数量大于第五阈值,基于所述第二图像特征对应的关键帧图像的特征点的三维位置信息和所述第一图像特征对应的关键帧图像的特征点的二维位置信息,确定所述第二位置信息。
在上述装置中,所述装置还包括:
第一选择模块,配置为从样本图像库中,选择满足预设条件的关键帧图像,得到关键帧图像集合;
第一提取模块,配置为提取每一关键帧图像的图像特征,得到关键图像特征集合;
第二提取模块,配置为提取样本图像的特征点,得到包含不同的特征点的样本特征点集合;
第二确定模块,配置为确定每一样本特征点,在所述关键帧图像的中所占的比值,得到比值向量集合;
第一存储模块,配置为存储所述比值向量集合和所述关键图像特征集合,得到所述第一地图。
在上述装置中,所述装置还包括:
第二选择模块,配置为从所述样本图像中选择预设数量的角点;
第三确定模块,配置为如果采集时间相邻的两个样本图像中包含的相同的角点数量大于等于第六阈值,确定所述样本图像对应的场景为连续场景;
第四确定模块,配置为如果采集时间相邻的两个样本图像中包含的相同的角点数量小于第六阈值,确定所述样本图像对应的场景为离散场景。
在上述装置中,所述第一选择模块,包括:
第一选择子模块,配置为如果所述样本图像对应的场景为离散场景,根据输入的选择指令,从样本图像库中选择关键帧图像;
第二选择子模块,配置为如果所述样本图像对应的场景为连续场景,根据预设的帧率或视差,从样本图像库中选择关键帧图像。
在上述装置中,所述第二确定模块,包括:
第二确定子模块,配置为根据样本图像库中包含的样本图像的第一数量和第i个样本特征点在样本图像库中出现的第一次数,确定第一平均次数;其中,i为大于等于1的整数;所述第一平均次数配置为表明所述第i个样本特征点平均在每一样本图像中出现的次数;
第三确定子模块,配置为根据所述第i个样本特征点在第j个关键帧图像中出现的第二次数和所述第j个关键帧图像中包含的样本特征点的第二数量,确定第二平均次数;其中,j为大于等于1的整数;所述第二平均次数配置为表明所述第i个样本特征点占据第j个关键帧图像中包含的样本特征点的比例;
第四确定子模块,配置为根据所述第一平均次数和所述第二平均次数,得到样本特征点在所述关键帧图像的中所占的比值,得到所述比值向量集合。
本申请实施例提供一种终端,包括存储器和处理器,所述存储器存储有可在处理器上运行的计算机程序,所述处理器执行所述程序时实现上述地图更新方法中的步骤。
本申请实施例提供一种计算机可读存储介质,其上存储有计算机程序,该计算机程序被处理器执行时实现上述地图更新方法中的步骤。
本申请实施例提供一种地图更新方法及装置、终端、存储介质,其中,首先,获取第一地图中的第一图像特征;然后,根据所述第一图像特征,从第二地图匹配出第二图像特征;最后,根据所述第一图像特征和所述第二图像特征,将所述第一地图中每一关键帧图像添加到所述第二地图中,得到更新的第二地图;如此,通过提取局部地图第一地图中的关键帧图像的图像特征,然后将该图像特征,与另一地图中关键帧图像的图像特征进行匹配,即可得到第二图像特征对应的关键帧图像,从而基于不同地图中相匹配的图像特征,将多个地图进行融合,从而实现了地图的更新,提高了更新后的地图的精度高。
附图说明
图1A为本申请实施例地图更新方法实现流程示意图;
图1B为本申请实施例地图更新方法的另一实现流程示意图;
图2A为本申请实施例地图更新方法的实现流程示意图;
图2B为本申请实施例创建预设地图的又一实现流程示意图;
图2C为本申请实施例地图更新方法另一实现流程示意图;
图3为本申请实施例地图更新方法的又一实现流程示意图;
图4为本申请实施例比值向量的结构示意图;
图5A为本申请实施例确定第二图像特征对应的关键帧图像的应用场景图;
图5B为本申请实施例确定采集设备的位置信息的结构示意图;
图6为本申请实施例地图更新装置的组成结构示意图。
具体实施方式
下面将结合本申请实施例中的附图,对本申请实施例中的技术方案进行清楚、完整地描述。
本申请实施例提供一种地图更新方法,图1A为本申请实施例地图更新方法实现流程示意图,如图1A所示,所述方法包括以下步骤:
步骤S101,获取第一地图中的第一图像特征。
这里,所述第一地图可以理解为是全局地图的一部分,比如,全局地图是包含了100个室内场景对应的关键帧图像,那么第一地图可以是包含了其中一部分室内场景对应的关键帧图像。第一图像特征包括:所所述第一地图中的关键帧图像的特征点的2D位置信息、3D位置信息和标识信息。在步骤S101中,首先,提取所述第一地图中的每一关键帧图像的特征点,得到特征点集合;然后,确定所述特征点集合中每一特征点的标识信息和每一所述特征点在所述关键帧图像中的2D位置信息;其中,特征点的标识信息可以理解为是能够唯一标识该特征点的描述子信息;最后,将每一2D位置信息分别映射在所述第一地图所处的坐标系中,得到每一所述特征点的3D位置信息。
步骤S102,根据所述第一图像特征,从第二地图匹配出第二图像特征。
这里,第一地图和第二地图为不同的两个地图,第一地图和所述第二地图对应的场景信息部分相同,这样保证能够从第二地图中匹配出与第一地图中的关键帧图像的第一图像特征相似度较高的第二图像特征;所述第二地图,也可以理解为是全局地图的一部分,比如,全局地图是包含了100个室内场景对应的关键帧图像,那么第二地图可以是包含了其中一部分室内场景对应的关键帧图像。第一地图和所述第二地图对应的场景信息至少部分相同;即,第一地图和第二地图中有少许的重叠部分。所述第二图像特征包括:所述第二图像特征包括:所述第二地图中的关键帧图像的特征点的2D位置信息、3D位置信息和标识信息;其中,所述3D位置信息是将所述2D位置信息映射在所述2D位置信息所处的坐标系中得到的。所述步骤S102可以理解为,从第二地图中存储的关键帧图像的图像特征中,选择与第一图像特征匹配度较高的第二图像特征。
步骤S103,根据所述第一图像特征和所述第二图像特征,将所述第一地图中每一关键帧图像添加到所述第二地图中,得到更新的第二地图。
这里,基于所述第二图像特征对应的关键帧图像的特征点的3D位置信息和所述第一图像特征对应的关键帧图像的特征点的2D位置信息,确定第一图像特征对应的关键帧图像的图像采集设备的位置信息。比如,首先,将第一图像特征对应的关键帧图像的特征点的2D位置信息转换为3D位置信息,然后,将该3D位置信息与第二地图所处的第二坐标系中的关键帧图像的特征点的3D位置信息,进行比对,以确定第一图像特征对应的关键帧图像的图像采集设备的位置信息。这样,同时考虑了特征点的2D位置信息和3D位置信息,那么当对第一图像特征对应的关键帧图像进行定位时,既可以得到第一图像特征对应的待匹配关键帧图像的图像采集设备的2D位置信息,还可以得到第一图像特征对应的关键帧图像的图像采集设备的3D位置信息,也可以理解为,既可以得到图像采集设备像的平面空间位置,还可以得到图像采集设备的立体空间位置;如此,依据这些丰富的位置信息,可以更加精确的将多个局部地图融合在一起。
在本申请实施例中,对于第一地图中的任意关键帧图像,通过提取图像特征,首先,从第二地图中找出与该图像特征匹配的第二图像特征,其次,基于两个图像特征的特征点的位置信息,最后,基于图像特征的位置信息和两个地图分别对应的坐标系之间的转换关系,将第一地图中的每一关键帧图像添加到第二地图中,从而完成对第二地图的更新,保证了良好的地图融合精度。
本申请实施例提供一种地图更新方法,图1B为本申请实施例地图更新方法的另一实现流程示意图,如图1B所示,所述方法包括以下步骤:
步骤S121,提取所述第一地图中的待匹配关键帧图像,得到第一图像特征。
这里,所述待匹配关键帧图像为所述第一地图中的关键帧图像。
步骤S122,根据所述第一图像特征,从第二地图中的关键帧图像的图像特征中,匹配出第二图像特征。
这里,采用预设的词袋模型,从第二地图中存储的关键帧图像的图像特征中,检索出与第一图像特征相似度较高的第二图像特征。
步骤S123,获取用于采集所述待匹配关键帧图像的图像采集设备在所述第一地图所处的第一坐标系中的第一位置信息。
这里,第一坐标系可以是三维坐标系,第一位置信息可以理解为,待匹配关键帧图像的图像采集设备在第一坐标系中的三维坐标值。
步骤S124,根据所述第一图像特征和所述第二图像特征,确定所述图像采集设备在所述第二地图所处的第二坐标系中的第二位置信息。
这里,首先得到第二地图所处的第二坐标系,第二坐标系为三维坐标系,然后,基于所述第二图像特征对应的关键帧图像的特征点的3D位置信息和所述第一图像特征对应的关键帧图像的特征点的2D位置信息,确定第二位置信息。第二位置信息可以理解为是,待匹配关键帧图像的图像采集设备在第二坐标系中的三维坐标值。比如,将第二图像特征对应的关键帧图像的特征点的3D位置信息和所述第一图像特征对应的特征点的2D位置信息,作为前端位姿跟踪算法(Perspectives-n-Point,PnP)算法的输入,先求出待匹配关键帧图像中特征点的2D位置信息(比如,2D坐标)在第一坐标系下该特征点的3D位置信息(比如,3D坐标),然后根据第二坐标系下第二特征对应的的特征点的3D位置信息和第一坐标系下的待匹配关键帧图像中特征点的3D位置信息,即可求解待匹配关键帧图像的图像采集设备的位置信息。
步骤S125,根据所述第一位置信息和所述第二位置信息,将所述第一地图中每一关键帧图像添加到所述第二地图中,得到更新的第二地图。
这里,更新的第二地图可以是全局地图,还可以是全局地图的一部分。所述步骤S125可以通过以下过程实现:
第一步,根据所述第一位置信息和所述第二位置信息,确定所述第一坐标系和所述第二坐标系之间的转换关系。
这里,第一坐标系和所述第二坐标系之间的转换关系,可以是第一坐标系相对于第二坐标系的旋转矩阵和平移向量。
第二步,据所述转换关系,对所述第一地图中的每一关键帧图像对应的图像采集设备在第一坐标系中的坐标进行调整,使得所述每一关键帧图像对应的图像采集设备的调整后的坐标与所述第二坐标系相匹配。
这里,据所述转换关系,对所述第一地图中的每一关键帧图像对应的图像采集设备在第一坐标系中的坐标进行调整的过程可以是:首先,利用旋转矩阵,对每一关键帧图像对应的图像采集设备在第一坐标系中的坐标,进行旋转;然后利用平移向量,对每一关键帧图像对应的图像采集设备在第一坐标系中的旋转后的坐标,进行平移,得到每一关键帧图像对应的图像采集设备的调整后的坐标。
第三步,将每一坐标调整后的图像采集设备对应的关键帧图像,添加到所述第二地图中,得到所述更新的第二地图。
这里,由于每一关键帧图像对应的图像采集设备在第一坐标系中的坐标均已调整为与第二坐标系相匹配的坐标,所以可以基于此,将第一地图中的每一关键帧图像添加到第二地图中,实现两个地图的融合,得到更新的第二地图(即全局地图)。在其他实施例中,还可以是将三个或三个以上的地图进行融合,得到全局地图。
上述步骤S123至步骤S125给出了一种实现“根据所述第一图像特征和所述第二图像特征,将所述第一地图中每一关键帧图像添加到所述第二地图中,得到更新的第二地图”的方式,在该方式中,基于待匹配关键帧的图像采集设备分别在第一坐标系和第二坐标系中的位置信息,确定第一坐标系和第二坐标系之间的转换关系,然后,将第一地图中每一关键帧图像对应的图像采集设备在第一坐标系中的坐标,按照该转换关系进行调整,这样第一地图中每一关键帧图像对应的图像采集设备在第一坐标系中的坐标与第二坐标系相匹配,从而将调整后的图像采集设备对应的关键帧图像添加到第二地图中,这样将两个地图精确的融合在一起。
在本申请实施例中,通过两个布局地图中的关键帧图像的图像特征,对多个局部地图进行融合,达成地图更新的目的,使得更新后的地图融合精度较高,鲁棒性强。
本申请实施例提供一种地图更新方法,图2A为本申请实施例地图更新方法的实现流程示意图,如图2A所示,所述方法包括以下步骤:
步骤S201,提取所述第一地图中的每一关键帧图像的特征点,得到特征点集合。
这里,对第一地图中的每一关键帧图像的特征点进行提取,得到特征点集合。
步骤S202,确定所述特征点集合中每一特征点的标识信息和每一所述特征点在所述关键帧图像中的2D位置信息。
这里,对于特征点集合中的每一特征点,确定该特征点的描述子信息即(标识信息),2D位置信息可以认为是该特征点的2D坐标。
步骤S203,将每一2D位置信息分别映射在所述第一地图所处的坐标系中,得到每一所述特征点的3D位置信息。
上述步骤S201至步骤S203给出了一种实现“对第一地图中的关键帧图像进行提取,得到第一图像特征”的方式,在该方式中,得到第一地图中的关键帧图像的每一特征点的2D坐标、该特征点的描述子信息和3D坐标。
步骤S204,分别确定不同的样本特征点在所述特征点集合中所占的比值,得到第一比值向量。
这里,第一比值向量可以是预先存储在第一地图对应的预设词袋模型中的。当需要对第一地图中的待匹配关键帧图像的图像特征进行匹配时,从该预设词袋模型中,获取第一比值向量。所述多个样本特征点互不相同。所述预设的词袋模型中包含多个不同的样本特征点和多个样本特征点在所述关键帧图像中包含的特征点中所占的比值。所述第一比值向量可以是根据样本图像数量、样本特征点在样本图像中出现的次数、样本特征点在待匹配关键帧图像里出现的次数和待匹配关键帧图像中出现的样本特征点的总数来确定;如公式(1)所示:
Figure PCTCN2020096505-appb-000001
Figure PCTCN2020096505-appb-000002
其中,N为样本图像数量(即第一数量),n i为样本特征点w i在样本图像中出现的次数(即第一次数),I t为t时刻采集的图像I,
Figure PCTCN2020096505-appb-000003
为样本特征点w i在时刻采集到的关键帧图像I t里出现的次数(即第二次数),
Figure PCTCN2020096505-appb-000004
为关键帧图像I t里出现的样本特征点总数(即第二数量)。通过样本特征点评分,得到每个关键帧图像的w维的浮点数向量,即比值向量,还可以将该比值向量作为预设的词袋模型的特征信息。
步骤S205,获取第二比值向量。
这里,所述第二比值向量为所述多个样本特征点在所述第二地图中的关键帧图像中包含的特征点中所占的比值;第二比值向量是预先存储在预设的词袋模型中的,所以当需要对待匹配关键帧图像的图像特征进行匹配时,从预设的词袋模型中获取该第二比值向量。第二比值向量的确定过程与第一比值向量的确定过程类似,均可采用公式(1)进行确定;而且所述第一比值向量和所述第二比值向量的维数相同。
步骤S206,根据所述第一图像特征、所述第一比值向量和所述第二比值向量,从所述关键帧图像的图像特征中,匹配出第二图像特征。
这里,所述步骤S205可以通过以下过程实现:
第一步,根据所述第一比值向量和所述第二比值向量,从所述第二地图中的关键帧图像的图像特征中,确定与所述第一图像特征的相似度大于第二阈值的相似图像特征。
这里,逐一的比较待匹配关键帧图像的第一比值向量v 1与第二地图中每一关键帧图像的第二比值向量v 2,采用这两个比值向量进行如公式(2)所示的计算,即可确定第二地图中每一关键帧图像与待匹配关键帧图像的相似度,从而筛选出相似度大于等于第二阈值的相似关键帧图像,得到相似关键帧图像集合。
第二步,确定所述相似图像特征所属的相似关键帧图像,得到相似关键帧图像集合。
第三步,从所述相似关键帧图像的图像特征中,选择与所述第一图像特征相似度满足预设相似度阈值的第二图像特征。
这里,从相似关键帧图像包含的图像特征中,选择与第一图像特征相似度最高的第二图像特征; 比如,首先,确定至少两个所述相似关键帧图像的采集时间之间的时间差,和所述至少两个相似关键帧图像的图像特征分别与所述第一图像特征的相似度差;然后,将所述时间差小于第三阈值,且所述相似度差小于第四阈值的相似关键帧图像进行联合,得到联合帧图像;也就是说,选择的是采集时间靠近,且与待匹配关键帧图像的相似度靠近的多个相似关键帧图像,说明这些关键帧图像可能是连续的的画面,所以将这样的多个相似关键帧图像联合在一起,组成联合帧图像(也可以成为岛),这样得到多个联合帧图像;最后,从所述联合帧图像的图像特征中,选择与所述第一图像特征相似度满足预设相似度阈值的第二图像特征。比如,先是分别确定多个联合帧图像中包含的每一关键帧图像的图像特征与所述第一图像特征的相似度之和;这样,逐一的确定多个联合帧图像中包含的多个关键帧图像的图像特征与第一图像特征的相似度之和。再,将相似度之和最大的联合帧图像,确定为与所述待匹配关键帧图像的相似度最高的目标联合帧图像;最后,根据目标联合帧图像的特征点的标识信息和所述待匹配关键帧图像的特征点的标识信息,从所述目标联合帧图像的图像特征中,选择与所述第一图像特征相似度满足预设相似度阈值的第二图像特征。这样,由于目标联合帧图像的特征点的标识信息和所述待匹配关键帧图像的特征点的标识信息,分别能够唯一的标识目标联合帧图像的特征点和待匹配关键帧图像的特征点,所以基于这两个标识信息,可以非常准确的从所述目标联合帧图像的图像特征中,选择与第一图像特征相似度最高的第二图像特征。从而保证了,为待匹配关键帧图像的第一图像特征匹配第二图像特征的准确度,保证了选择到的第二图像特征与第一图像特征的相似度极高。在本实施例中,包含该第二图像特征的关键帧图像,说明该关键帧图像与待匹配关键帧图像非常相似,可以认为是与该待匹配关键帧图像最相似的图像。
上述步骤S204至步骤S206给出了一种实现“根据所述第一图像特征,从第二地图中的关键帧图像的图像特征中,匹配出第二图像特征,”的方式,在该方式中,通过采用预设的词袋模型从第二地图中的关键帧图像的图像特征中检索出与第一图像特征的相匹配的第二图像特征,保证了第二图像特征与第一图像特征的相似度。
步骤S207,确定所述第二图像特征对应的关键帧图像中包含的任意两个特征点之间,小于第一阈值的目标欧式距离,得到目标欧式距离集合。
这里,首先,确定第二图像特征对应的关键帧图像中包含的任意两个特征点之间的欧式距离,然后,从中选择小于第一阈值的欧式距离,作为目标欧式距离,以得到目标欧式距离集合;这是对于待匹配关键帧图像中的一个特征点进行处理,可得到一个目标欧式距离集合,那么对于待匹配关键帧图像中的多个特征点进行处理,则可得到多个欧式距离集合。所述小于第一阈值的目标欧式距离,还可以认为是首先从多个欧式距离中确定最小的欧式距离,然后判断该最小的欧式距离是否小于第一阈值,若小于,则确定该最小的欧式距离为目标欧式距离,那么目标欧式距离集合也就是多个欧式距离集合中,欧式距离最小的一个集合。
步骤S208,如果所述目标欧式距离集合中包含的目标欧式距离的数量大于第五阈值,基于所述第二图像特征对应的关键帧图像的特征点的3D位置信息和所述第一图像特征对应的关键帧图像的特征点的2D位置信息,确定所述第二位置信息。
这里,如果目标欧式距离集合中包含的目标欧式距离的数量大于第五阈值,说明目标欧式距离的数量是足够大,也说明与第一图像特征相匹配的特征点足够多,说明这个包含第二图像特征的关键帧图像与待匹配关键帧图像的相似度足够高。然后,将第二图像特征的关键帧图像的特征点的3D位置信息和所述第一图像特征对应的待匹配关键帧图像的特征点的2D位置信息,作为PnP算法的输入,先求出待匹配关键帧图像的当前帧中特征点的2D坐标,在当前坐标系下该特征点的3D坐标,然后根据地图坐标系下的关键帧图像的特征点的3D位置信息和当前坐标系下的待匹配关键帧图像的当前帧中特征点的3D位置信息,即可求解图像采集设备的位置信息。
上述步骤S206至步骤S208给出了一种实现“根据所述第一图像特征和所述第二图像特征,确定所述图像采集设备在所述第二地图所处的第二坐标系中的第二位置信息”的方式,在该方式中,同时考虑关键帧图像的2D和3D位置信息,在定位结果上可以同时提供位置和姿态,所以提高了确定图像采集设备的位置的准确度,从而能够有效的对第一地图中关键帧图像的图像采集设备的坐标,进行转换,更好的将第一地图中的关键帧图像融合到第二地图中,保证了地图的融合精度和鲁棒性。
步骤S209,获取用于采集所述待匹配关键帧图像的图像采集设备在所述第一地图所处的第一坐标系中的第一位置信息。
这里,在第一地图中每一关键帧图像对应的图像采集设备均有一个第一位置信息。第一位置信息可以理解为该图像采集设备在第一坐标系中的坐标值。
步骤S210,根据所述第一位置信息和所述第二位置信息,将所述第一地图中每一关键帧图像添 加到所述第二地图中,得到更新的第二地图。
在本申请实施例中,是通过图像采集设备,得到待匹配关键帧图像,加载构建好的预设地图,并利用预设的词袋模型检索匹配到待匹配关键帧图像相对应的第二图像特征对应的关键帧图像,最后,再将待匹配关键帧图像的特征点的2D位置信息和关键帧图像的特征点的3D位置信息,作为PnP算法的输入,以得到待匹配关键帧图像的当前相机在第二地图中的位置,这样将第一地图中的关键帧图像对应的图像采集设备的坐标均转换为与第二地图的第二坐标系相匹配的坐标,然后再将第一地图中的关键帧图像精确的融合到第二地图中;这样,通过关键帧图像即可确定第一地图中的关键帧图像对应的图像采集设备,在第二地图的第二坐标系下的坐标,从而更加准确的将两个地图融合在一起,得到更新的第二地图,不需要两个局部地图有大量的重复区域,也能保证良好的地图融合精度,在以众包进行地图更新或多人创建地图的时候,提升了地图融合的稳定性,同时也提升了构建局部地图的效率。
本申请实施例提供一种地图更新方法,图2B为本申请实施例创建预设地图的又一实现流程示意图,如图2B所示,所述方法包括以下步骤:
步骤S221,从样本图像库中,选择满足预设条件的关键帧图像,得到关键帧图像集合。
这里,首先,确定该样本图像对应的场景为连续场景还是离散场景,如果是离散场景,过程如下:
第一步,从所述样本图像中选择预设数量的角点;所述角点为所述样本图像中与周围预设数量的像素点具有较大差别的像素点;比如,选择150个角点。
第二步,如果采集时间相邻的两个样本图像中包含的相同的角点数量大于等于第六阈值,确定所述样本图像对应的场景为连续场景;两个样本图像的采集时间相邻,还可以理解为是连续的两个样本图像,判断这两个样本图像中包含的相同的角点的数量,数量越大,说明这两个样本图像的相关度越高,也说明这两个样本图像是来自于连续场景的图像。连续场景,比如,单一的室内环境,比如,卧室、客厅或单个会议室等。
第三步,如果采集时间相邻的两个样本图像中包含的相同的角点数量小于第六阈值,确定所述样本图像对应的场景为离散场景。这两个样本图像中包含的相同的角点的数量越小,说明这两个样本图像的相关度越低,也说明这两个样本图像是来自于离散场景的图像。离散场景,比如,在多个室内环境下,比如,一栋楼里的多个房间或者一层里的多个会议室等。
然后,如果样本图像对应的场景为离散场景,根据输入的选择指令,从样本图像库中选择关键帧图像;即,如果样本图像属于离散场景,说明多个样本图像对应的不是一个场景,那么用户手动选择关键帧图像,这样,保证了不同的环境下,所选的关键图像的有效性。
如果样本图像对应的场景为连续场景,根据预设的帧率或视差,从样本图像库中选择关键帧图像;即,如果样本图像属于连续场景,说明多个样本图像对应的是同一个场景,那么通过事先设置预设的帧率或者预设的视差,自动选择满足该预设的帧率或者预设的视差的样本图像作为关键帧图像,这样,既所选的关键图像的有效性,还提高了选择关键帧图像的效率。
步骤S222,提取每一关键帧图像的图像特征,得到关键图像特征集合。
这里,关键帧的图像特征包括:关键帧图像的特征点的2D位置信息、3D位置信息和能够唯一标识该特征点的标识信息。得到关键图像特征集合,以便于从关键图像特征集合中匹配出与第一图像特征高度相似的第二图像特征,从而得到相应的第二图像特征对应的关键帧图像。
步骤S223,确定每一样本特征点,在关键帧图像的中所占的比值,得到比值向量集合。
这里,得到比值向量集合之后,将不同的样本特征点和该比值向量集合存储于预设的词袋模型中,以便于采用预设的词袋模型从关键帧图像中检索出待匹配关键帧图像的第二图像特征对应的关键帧图像。所述步骤S223可以通过以下过程实现:
首先,根据样本图像库中包含的样本图像的第一数量和第i个样本特征点在样本图像库中出现的第一次数,确定第一平均次数。i为大于等于1的整数;第一平均次数用于表明所述第i个样本特征点平均在每一样本图像中出现的次数;比如,样本图像的第一数量为N,第i个样本特征点在样本图像库中出现的第一次数为n i,通过公式(1)即可得到第一平均次数idf(i)。
其次,根据所述第i个样本特征点在第j个关键帧图像中出现的第二次数和所述第j个关键帧图像中包含的样本特征点的第二数量,确定第二平均次数;j为大于等于1的整数;第二平均次数配置为表明所述第i个样本特征点占据第j个关键帧图像中包含的样本特征点的比例;比如,第二次数为
Figure PCTCN2020096505-appb-000005
第二数量为
Figure PCTCN2020096505-appb-000006
通过公式(1)即可得到第二平均次数tf(i,I t)。
最后,根据所述第一平均次数和所述第二平均次数,得到样本特征点在关键帧图像的中所占的比值,得到所述比值向量集合。比如,根据公式(1),将第一平均次数与第二平均次数相乘,即可得到比值向量
Figure PCTCN2020096505-appb-000007
步骤S224,存储所述比值向量集合和所述关键图像特征集合,得到所述第一地图。
这里,将关键帧图像对应的比值向量集合和关键图像特征集合存储在第一地图中,以便于对图像采集设备进行定位时,采用该比值向量集合与利用预设的词袋模型确定的待匹配关键帧图像对应的比值向量集合进行比对,以从关键图像特征集合中确定与待匹配关键帧图像高度相似的第二图像特征对应的关键帧图像。
上述步骤S221至步骤S224给出了构建局部地图,比如第一地图和第二地图的过程,重复上述步骤S221至步骤S224,可得到第二地图,还可以得到任意场景对应的地图。
在本申请实施例中,对于样本图像的离散或连续场景,采用不同的关键帧图像选取方式,保证了所选择的关键帧图像的有效性,然后,从关键帧图像中提取图像特征,构建第一地图,保证了第一地图的准确性。
本申请实施例提供一种地图更新方法,图2C为本申请实施例地图更新方法另一实现流程示意图,如图2C所示,所述方法包括以下步骤:
步骤S231,从样本图像库中,选择满足预设条件的关键帧图像,得到关键帧图像集合。
步骤S232,提取每一关键帧图像的图像特征,得到关键图像特征集合。
步骤S233,提取样本图像的特征点,得到包含不同的特征点的样本特征点集合。
步骤S234,确定每一样本特征点,在关键帧图像的中所占的比值,得到比值向量集合。
步骤S235,存储所述比值向量集合和所述关键图像特征集合,得到所述第一地图和第二地图。
上述步骤S231至步骤S235,完成了预设地图的创建过程,将关键帧图像的图像特征和比值向量集合存储在预设地图中,以便于能够根据比值向量集合从第二地图中的关键帧图像的图像特征中搜索出与待匹配关键帧图像的图像特征相匹配的第二图像特征。
步骤S236,从第一地图中的待匹配关键帧图像的图像特征中,提取待匹配关键帧图像的第一图像特征。
这里,当对图像采集设备进行定位时,需要先加载第一地图和第二地图。
步骤S237,根据所述第一图像特征,从第二地图中的关键帧图像的图像特征中,匹配出第二图像特征。
步骤S238,根据所述第一图像特征和所述第二图像特征,确定所述图像采集设备在所述第二地图所处的第二坐标系中的第二位置信息。
上述步骤S236至步骤S238,给出了实现对图像采集设备进行定位的过程,在该过程中,通过采用从预设地图中存储的关键帧图像中匹配出与第一图像特征高度相似的第二图像特征,然后,利用利用这两个图像特征中的2D位置信息和3D位置信息,即可最终确定采集设备的在第二坐标系中的第二位置信息。
步骤S239,获取用于采集所述待匹配关键帧图像的图像采集设备在所述第一地图所处的第一坐标系中的第一位置信息。
步骤S240,根据所述第一位置信息和所述第二位置信息,将所述第一地图中每一关键帧图像添加到所述第二地图中,得到更新的第二地图。
在本申请实施例中,同时采用关键帧图像的2D和3D位置信息,保证了对采集设备的定位结果的准确度,定位成功率高,鲁棒性强,这样,将第一地图中的关键帧图像融合到第二地图中,保保证了融合后的地图的融合精度。
本申请实施例提供一种地图更新方法,图3为本申请实施例地图更新方法的又一实现流程示意图,如图3所示,所述方法包括以下步骤:
步骤S301,针对样本图像所属的场景,选择不同的关键帧图像选取方式。
这里,样本图像所属的场景包括:离散场景或连续场景;关键帧图像的选取方法包括手动选择和自动选择两类,手动选择要求建图者手动选择需要纳入地图的关键帧图像,而自动选择是根据帧率或视差自动选择图像作为关键帧图像的方法。在针对关键帧图像的图像特征提取过程中,每个关键帧图像中都会提取出150个FAST特征角点,连续两个关键帧图像中拥有相同角点的比例定义为角点追踪率。本申请实施例将有序的关键帧图像序列并且平均角点追踪率大于30%的场景定义为连 续场景,否则即为离散场景。连续场景的关键帧图像选取方法使用自动选择法;而离散场景的关键帧图像选取方法使用手动选择法。连续场景适用于单一的室内环境下,比如卧室、客厅、单个会议室等;离散场景更适合在多个室内环境下使用,比如一栋楼里的多个房间,或者一层里的多个会议室等。在地图构建的过程中,连续场景和离散场景的关键帧图像选取策略不同,适用场景不同。这样针对室内的离散或连续场景,通过不同的关键帧图像的选取方式,提取图像特征进行地图构建,这样定位过程不依赖于外部基站设备,成本低,定位精度高,鲁棒性强。
步骤S302,利用摄像头进行关键帧图像采集。
这里,该摄像头可以是单目摄像头还可以是双目摄像头。
步骤S303,采集过程中实时提取关键帧图像中的图像特征。
这里,图像特征提取是对关键帧图像的一种解释和标注的过程。在步骤S303中,需要提取关键帧图像的特征点的2D位置信息、3D位置信息和标识信息(即该特征点的描述子信息);其中,关键帧图像的特征点的3D位置信息是将关键帧图像的特征点的2D位置信息映射在预设地图所处的三维坐标系中得到的。比如,对关键帧图像提取多个2D的特征点,提取数量为150个(150为经验值,特征点数量过少,跟踪失败率高,特征点数量过多,影响算法效率),用于图像跟踪;并对该特征点进行描述子的提取,用于特征点匹配;其次,通过三角化方法计算得到特征点的3D位置信息(即深度信息),用于确定采集相机的位置。
步骤S304,采集过程中实时确定每一样本特征点,在关键帧图像的中所占的比值,得到比值向量。
这里,步骤S304可以理解为,在关键帧图像的采集过程中,针对当前帧图像,实时提取该关键帧图像的的比值向量,如图4所示,用词汇树的形式来描述词袋模型,词袋模型中包括样本图像库41,即词汇树的根结点;样本图像42、43和44,即叶子结点42、43;样本特征点1至3为样本图像42中不同的样本特征点,本特征点4至6为样本图像43中不同的样本特征点,本特征点7至9为样本图像44中不同的样本特征点。在词袋模型中假设有w种样本特征点,即w为词袋模型的样本图像里提取出来的特征点种类数量。所以词袋模型里一共有w个样本特征点。每个样本特征点会对该关键帧图像进行评分,评分值为0~1的浮点数,这样每个关键帧图像都可以用w维的浮点数来表示,这个w维向量就是词袋模型输出的比值向量
Figure PCTCN2020096505-appb-000008
评分的过程,如公式(1)所示。
上述步骤S301至步骤S304,构建出一张依赖于关键帧图像的离线的局部地图,即第一地图和第二地图,该第一地图和第二地图以二进制格式存储关键帧图像的图像特征(包括:2D位置信息、2D位置信息和标识信息,比如,2D坐标、3D坐标、和描述子信息)到本地设备,当需要对图像采集设备进行时,该第一地图和第二地图将被加载使用。
步骤S305,加载构建好的第一地图和第二地图。
这里,以第二地图的第二坐标系作为全局坐标系。
步骤S306,提取第一地图中的待匹配关键帧图像的图像特征,得到第一图像特征。
这里,待匹配关键帧图像为第一地图中的关键帧图像中的任一个。
步骤S307,通过词袋模型检索待匹配关键帧图像的当前帧在第二地图中的第二图像特征对应的关键帧图像。
这里,所述通过词袋模型检索待匹配关键帧图像的当前帧在第二地图中的第二图像特征对应的关键帧图像,可以理解为利用词袋模型的特征信息即比值向量集合,进行检索待匹配关键帧图像的当前帧在第二地图中的第二图像特征对应的关键帧图像。
所述步骤S307可以通过以下过程实现:
第一步,查找待匹配关键帧图像的当前帧和第二地图中的每个关键帧图像的相似度,相似度s(v 1,v 2)的计算方式如公式(2)所示。
Figure PCTCN2020096505-appb-000009
其中,v 1和v 2分别表示词袋模型中包含的每一样本特征点在所述待匹配关键帧图像的当前帧中所占的第一比值向量,和每一样本特征点在第二地图中的关键帧图像中所占的第二比值向量。如果词袋模型中包含w种样本特征点,那么第一比值向量和第二比值向量均为w维的向量。通过采用筛选出第二地图中的关键帧图像中相似度达到第二阈值的相似关键帧图像,成为相似关键帧图像集合。
第二步,在相似关键帧图像集合选取时间戳之差小于第三阈值,且相似度差小于第四阈值的相 似关键帧图像联合在一起,得到联合帧图像(或被称为岛)。
这里,第二步可以理解为在相似关键帧图像集合选取时间戳靠近,且相似度的匹配分数靠近的相似关键帧图像联合在一起,被成为岛;这样将相似关键帧图像集合就被划分成了多联合帧图像(即多个岛)。联合帧图像中的第一个关键帧图像与最后一个关键帧图像之间的相似度之比非常小,该相似度之比
Figure PCTCN2020096505-appb-000010
如公式(3)所示:
Figure PCTCN2020096505-appb-000011
其中,
Figure PCTCN2020096505-appb-000012
和s(v t,v t-△t)分别表示一前一后两个关键帧图像的与当前帧的待匹配关键帧图像的相似度。
第三步,分别确定多个联合帧图像中包含的每一关键帧图像的图像特征与所述第一图像特征的相似度之和,如公式(4)所示,
Figure PCTCN2020096505-appb-000013
第四步,将相似度之和最大的联合帧图像,确定为与所述待匹配关键帧图像的相似度最高的目标联合帧图像,从所述目标联合帧图像中找出与待匹配关键帧图像的当前帧相似度最高的第二图像特征对应的关键帧图像。
步骤S308,采用PnP算法,确定待匹配关键帧图像对应的图像采集设备在第二地图所处的第二坐标系中的第二位置信息在地图坐标系中的位置信息。
这里,所述步骤S308可以通过以下步骤实现:
第一步,对待匹配关键帧图像的当前帧X C的第N个特征点F CN,遍历第二图像特征对应的关键帧图像X 3的所有特征点,并确定第二图像特征对应的关键帧图像中任意两个特征点之间的欧式距离。如图5A所示,待匹配关键帧图像的当前帧Xc51,与该当前帧Xc51匹配的第二图像特征对应的关键帧图像X352。计算特征点X053和X154之间的欧式距离,得到欧式距离F0501;计算特征点X154和X255之间的欧式距离,得到欧式距离F1502;计算特征点X255和X352之间的欧式距离,得到欧式距离F2503;计算特征点Xc51和X456之间的欧式距离,得到欧式距离F3504。
第二步,选择欧式距离最小的一组(即目标欧式距离集合)进行阈值判断,若小于第一阈值,确定为目标欧式距离,则形成目标欧式距离集合,否则不形成目标欧式距离集合,跳转至第一步,直至遍历X C的所有特征点,进入第三步。比如,如图5A所示,通过比较多个欧式距离,得到一组最小的欧式距离组合{F 1,F 2,F 3}。
第三步,形成目标欧式距离集合,可表示为{F 1,F 2,F 3},若目标欧式距离集合的元素数量大于第五阈值,则进行第四步,否则算法结束,输出匹配帧X 3的位置信息。
第四步,基于目标欧式距离集合,调用PnP中的函数求解出X C在地图坐标系下的位置信息。其中,PnP算法的过程如下:
PnP算法的输入是包含第二图像特征的关键帧图像中的特征点的3D坐标和待匹配关键帧图像的当前帧中特征点的2D坐标,该算法的输出是待匹配关键帧图像的当前帧对应的图像采集设备在第二地图的第二坐标系中的位置。
PnP算法不是直接根据匹配对序列求出相机位姿矩阵的,而是先求出待匹配关键帧图像的当前帧中特征点的2D坐标在当前坐标系下待匹配关键帧图像的当前帧中特征点的3D坐标,然后根据地图坐标系下的3D坐标系和当前坐标系下的待匹配关键帧图像的当前帧中特征点的3D坐标求解相机位姿的。PnP算法的求解是从余弦定理开始的,设当前坐标系中心为点O,A、B和C为待匹配关键帧图像的当前帧中三个特征点,如图5B所示:
根据余弦定理,A、B和C之间的关系如公式(5)所示:
OA 2+OB 2-2·OA·OB·cos<a,b>=AB 2
OA 2+OC 2-2·OA·OC·cos<a,c>=AC 2
OB 2+OC 2-2·OB·OC·cos<b,c>=BC 2       (5);
对上式进行消元,同时除以OC 2,并另
Figure PCTCN2020096505-appb-000014
则可得公式(6):
Figure PCTCN2020096505-appb-000015
接着进行替换,另
Figure PCTCN2020096505-appb-000016
则可得公式(7):
x 2+y 2-2·x·y·cos<a,b>=u 2
x 2+1-2·x·cos<a,c>=wu
y 2+1-2·y·cos<b,c>=vu      (7);
将公式(4)分别代入公式(6)和(7),则分别得到公式(8)和(9):
(1-w)x 2-w·y 2-2·x·cos<a,c>+2·w·x·y·cos<a,b>+1=0  (8);
(1-v)y 2-v·x 2-2·y·cos<b,c>+2·v·x·y·cos<a,b>+1=0  (9);
其中,由于A、B和C的2D坐标是已知的,所以w,v,cos<a,c>,cos<b,c>,cos<a,b>都是已知量,因此,未知量只有x,y两个,通过公式(8)和(9)可以求得x,y的值,从而,可以求解OA、OB和OC的值,如公式(10)所示:
Figure PCTCN2020096505-appb-000017
最后,即可得到A、B和C三个特征点在当前三维坐标系下的3D坐标,分别可通过公式(11)得到:
Figure PCTCN2020096505-appb-000018
得到A、B和C三个特征点在当前三维坐标系下的3D坐标后,然后通过地图坐标系到当前坐标系的变换,确定采集设备的位置。
上述步骤S305至步骤S308,对于图像采集设备采集到的待匹配关键帧图像加载构建好的离线地图,通过词袋模型在第二地图中的关键帧图像中检索待匹配关键帧图像的第二图像特征对应的关键帧图像,最后采用PnP算法求解当前相机在第二地图中的精确位姿,以确定该设备在第二坐标系下的位置和姿态,从而能够更加准确的将第一地图中的关键帧图像添加到第二地图中,实现地图更新。
步骤S309,将第一地图中每一关键帧图像对应的图像采集设备在第一坐标系中的坐标,进行调整,以使调整后的坐标与第二坐标系匹配,并将坐标调整后的图像采集设备对应的关键帧图像添加到第二地图中,得到更新的第二地图。
这里,首先,确定该图像采集设备在第一坐标系中的第一位置信息,其次,确定该图像采集设备在第二坐标系中的第二位置信息;再次,基于第一位置信息和第二位置信息,确定第一坐标系相对于第二坐标系的旋转矩阵和平移向量;利用该旋转矩阵和平移向量对该图像采集设备在第一坐标系中的坐标进行调整,使得调整后的图像采集设备的坐标与第二坐标系匹配。
在步骤S309之后,反复执行步骤S305至步骤S309,将其他局部地图融合到第二坐标系中(即全局坐标系)中,完成全局地图的融合。
在申请实施例中,同时考虑关键帧图像的2D坐标和3D坐标,在定位结果中可以提供采集设备 的3D坐标,提高了图像采集设备的定位准确度,这样保证了将第一地图中的关键帧图融合到第二地图中的融合精度,使得更新的地图鲁棒性强。
本申请实施例提供一种地图更新装置,该装置包括所包括的各模块、以及各模块所包括的各单元,可以通过计算机设备中的处理器来实现;当然也可通过具体的逻辑电路实现;在实施的过程中,处理器可以为中央处理器(CPU)、微处理器(MPU)、数字信号处理器(DSP)或现场可编程门阵列(FPGA)等。
图6为本申请实施例地图更新装置的组成结构示意图,如图6所示,所述装置600包括:第一获取模块601、第一匹配模块602和第一更新模块603,其中:所述第一获取模块601,配置为获取第一地图中的第一图像特征;所述第一匹配模块602,配置为根据所述第一图像特征,从第二地图匹配出第二图像特征;所述第一更新模块603,配置为根据所述第一图像特征和所述第二图像特征,将所述第一地图中每一关键帧图像添加到所述第二地图中,得到更新的第二地图。
在上述装置中,所述第一获取模块601,包括:第一提取子模块,配置为提取所述第一地图中的待匹配关键帧图像,得到第一图像特征;其中,所述待匹配关键帧图像为所述第一地图中的关键帧图像;对应地,所述第一匹配模块602,包括:第一匹配子模块,配置为根据所述第一图像特征,从第二地图中的关键帧图像的图像特征中,匹配出所述第二图像特征。
在上述装置中,所述第一更新模块603,包括:第一获取子模块,配置为获取用于采集所述待匹配关键帧图像的图像采集设备在所述第一地图所处的第一坐标系中的第一位置信息;第一确定子模块,配置为根据所述第一图像特征和所述第二图像特征,确定所述图像采集设备在所述第二地图所处的第二坐标系中的第二位置信息;第一更新子模块,配置为根据所述第一位置信息和所述第二位置信息,将所述第一地图中每一关键帧图像添加到所述第二地图中,得到更新的第二地图。
在上述装置中,所述第一更新子模块,包括:第一确定单元,配置为根据所述第一位置信息和所述第二位置信息,确定所述第一坐标系和所述第二坐标系之间的转换关系;第一调整单元,配置为根据所述转换关系,对所述第一地图中的每一关键帧图像对应的图像采集设备在第一坐标系中的坐标进行调整,使得所述每一关键帧图像对应的图像采集设备的调整后的坐标与所述第二坐标系相匹配;第一添加单元,配置为将每一坐标调整后的图像采集设备对应的关键帧图像,添加到所述第二地图中,得到所述更新的第二地图。
在上述装置中,所述第一地图和所述第二地图对应的场景信息至少部分相同。
在上述装置中,所述第一图像特征包括:所述第一地图中的关键帧图像的特征点的2D位置信息、3D位置信息和标识信息;所述第二图像特征包括:所述第二地图中的关键帧图像的特征点的2D位置信息、3D位置信息和标识信息;其中,所述3D位置信息是将所述2D位置信息映射在所述2D位置信息所处的坐标系中得到的。
在上述装置中,所述第一提取子模块,包括:第一提取单元,配置为提取所述第一地图中的每一关键帧图像的特征点,得到特征点集合;第二确定单元,配置为确定所述特征点集合中每一特征点的标识信息和每一所述特征点在所述关键帧图像中的2D位置信息;第一映射单元,配置为将每一2D位置信息分别映射在所述第一地图所处的坐标系中,得到每一所述特征点的3D位置信息。
在上述装置中,所述第一匹配子模块,包括:第三确定单元,配置为分别确定不同的样本特征点在所述特征点集合中所占的比值,得到第一比值向量;第一获取单元,配置为获取第二比值向量,所述第二比值向量为所述多个样本特征点在所述第二地图中的关键帧图像中包含的特征点中所占的比值;第一匹配单元,配置为根据所述第一图像特征、所述第一比值向量和所述第二比值向量,从所述第二地图中的关键帧图像的图像特征中,匹配出第二图像特征。
在上述装置中,所述第一匹配单元,包括:第一确定子单元,配置为根据所述第一比值向量和所述第二比值向量,从所述第二地图中的关键帧图像的图像特征中,确定与所述第一图像特征的相似度大于第二阈值的相似图像特征;第二确定子单元,配置为确定所述相似图像特征所属的相似关键帧图像,得到相似关键帧图像集合;第一选择子单元,配置为从所述相似关键帧图像的图像特征中,选择与所述第一图像特征相似度满足预设相似度阈值的第二图像特征。
在上述装置中,所述第一选择子单元,配置为:确定至少两个所述相似关键帧图像的采集时间之间的时间差,和所述至少两个相似关键帧图像的图像特征分别与所述第一图像特征的相似度差;将所述时间差小于第三阈值,且所述相似度差小于第四阈值的相似关键帧图像进行联合,得到联合帧图像;从所述联合帧图像的图像特征中,选择与所述第一图像特征相似度满足预设相似度阈值的第二图像特征。
在上述装置中,所述第一选择子单元,配置为:分别确定多个联合帧图像中包含的每一关键帧 图像的图像特征与所述第一图像特征的相似度之和;将相似度之和最大的联合帧图像,确定为与所待匹配关键帧图像的相似度最高的目标联合帧图像;根据目标联合帧图像的特征点的标识信息和所述待匹配关键帧图像的特征点的标识信息,从所述目标联合帧图像的图像特征中,选择与所述第一图像特征相似度满足预设相似度阈值的第二图像特征。
在上述装置中,所述装置还包括:第一确定模块,配置为确定所述第二图像特征对应的关键帧图像中包含的任意两个特征点之间,小于第一阈值的目标欧式距离,得到目标欧式距离集合;对应地,所述第一确定子模块,包括:第四确定单元,配置为如果所述目标欧式距离集合中包含的目标欧式距离的数量大于第五阈值,基于所述第二图像特征对应的关键帧图像的特征点的3D位置信息和所述第一图像特征对应的关键帧图像的特征点的2D位置信息,确定所述第二位置信息。
在上述装置中,所述装置还包括:第一选择模块,配置为从样本图像库中,选择满足预设条件的关键帧图像,得到关键帧图像集合;第一提取模块,配置为提取每一关键帧图像的图像特征,得到关键图像特征集合;第二提取模块,配置为提取样本图像的特征点,得到包含不同的特征点的样本特征点集合;第二确定模块,配置为确定每一样本特征点,在所述关键帧图像的中所占的比值,得到比值向量集合;第一存储模块,配置为存储所述比值向量集合和所述关键图像特征集合,得到所述第一地图。
在上述装置中,所述装置还包括:第二选择模块,配置为从所述样本图像中选择预设数量的角点;第三确定模块,配置为如果采集时间相邻的两个样本图像中包含的相同的角点数量大于等于第六阈值,确定所述样本图像对应的场景为连续场景;第四确定模块,配置为如果采集时间相邻的两个样本图像中包含的相同的角点数量小于第六阈值,确定所述样本图像对应的场景为离散场景。
在上述装置中,所述第一选择模块,包括:第一选择子模块,配置为如果所述样本图像对应的场景为离散场景,根据输入的选择指令,从样本图像库中选择关键帧图像;第二选择子模块,配置为如果所述样本图像对应的场景为连续场景,根据预设的帧率或视差,从样本图像库中选择关键帧图像。
在上述装置中,所述第二确定模块,包括:第二确定子模块,配置为根据样本图像库中包含的样本图像的第一数量和第i个样本特征点在样本图像库中出现的第一次数,确定第一平均次数;其中,i为大于等于1的整数;所述第一平均次数配置为表明所述第i个样本特征点平均在每一样本图像中出现的次数;第三确定子模块,配置为根据所述第i个样本特征点在第j个关键帧图像中出现的第二次数和所述第j个关键帧图像中包含的样本特征点的第二数量,确定第二平均次数;其中,j为大于等于1的整数;所述第二平均次数配置为表明所述第i个样本特征点占据第j个关键帧图像中包含的样本特征点的比例;第四确定子模块,配置为根据所述第一平均次数和所述第二平均次数,得到样本特征点在所述关键帧图像的中所占的比值,得到所述比值向量集合。
以上装置实施例的描述,与上述方法实施例的描述是类似的,具有同方法实施例相似的有益效果。对于本申请装置实施例中未披露的技术细节,请参照本申请方法实施例的描述而理解。
需要说明的是,本申请实施例中,如果以软件功能模块的形式实现上述的地图更新方法,并作为独立的产品销售或使用时,也可以存储在一个计算机可读取存储介质中。基于这样的理解,本申请实施例的技术方案本质上或者说对相关技术做出贡献的部分可以以软件产品的形式体现出来,该计算机软件产品存储在一个存储介质中,包括若干指令用以使得包含该存储介质的设备自动测试线执行本申请各个实施例所述方法的全部或部分。而前述的存储介质包括:U盘、移动硬盘、只读存储器(Read Only Memory,ROM)、磁碟或者光盘等各种可以存储程序代码的介质。
对应地,本申请实施例提供一种计算机可读存储介质,其上存储有计算机程序,该计算机程序被处理器执行时实现上述实施例中提供的地图更新方法中的步骤。这里需要指出的是:以上存储介质和设备实施例的描述,与上述方法实施例的描述是类似的,具有同方法实施例相似的有益效果。对于本申请存储介质和设备实施例中未披露的技术细节,请参照本申请方法实施例的描述而理解。应理解,说明书通篇中提到的“一个实施例”或“一实施例”意味着与实施例有关的特定特征、结构或特性包括在本申请的至少一个实施例中。因此,在整个说明书各处出现的“在一个实施例中”或“在一实施例中”未必一定指相同的实施例。此外,这些特定的特征、结构或特性可以任意适合的方式结合在一个或多个实施例中。应理解,在本申请的各种实施例中,上述各过程的序号的大小并不意味着执行顺序的先后,各过程的执行顺序应以其功能和内在逻辑确定,而不应对本申请实施例的实施过程构成任何限定。上述本申请实施例序号仅仅为了描述,不代表实施例的优劣。需要说明的是,在本文中,术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含,从而使得包括一系列要素的过程、方法、物品或者装置不仅包括那些要素,而且还包括没有明确列出的 其他要素,或者是还包括为这种过程、方法、物品或者装置所固有的要素。在没有更多限制的情况下,由语句“包括一个……”限定的要素,并不排除在包括该要素的过程、方法、物品或者装置中还存在另外的相同要素。
在本申请所提供的几个实施例中,应该理解到,所揭露的设备和方法,可以通过其它的方式实现。以上所描述的设备实施例仅仅是示意性的,例如,所述单元的划分,仅仅为一种逻辑功能划分,实际实现时可以有另外的划分方式,如:多个单元或组件可以结合,或可以集成到另一个系统,或一些特征可以忽略,或不执行。另外,所显示或讨论的各组成部分相互之间的耦合、或直接耦合、或通信连接可以是通过一些接口,设备或单元的间接耦合或通信连接,可以是电性的、机械的或其它形式的。上述作为分离部件说明的单元可以是、或也可以不是物理上分开的,作为单元显示的部件可以是、或也可以不是物理单元;既可以位于一个地方,也可以分布到多个网络单元上;可以根据实际的需要选择其中的部分或全部单元来实现本申请实施例方案的目的。另外,在本申请各实施例中的各功能单元可以全部集成在一个处理单元中,也可以是各单元分别单独作为一个单元,也可以两个或两个以上单元集成在一个单元中;上述集成的单元既可以采用硬件的形式实现,也可以采用硬件加软件功能单元的形式实现。
本领域普通技术人员可以理解:实现上述方法实施例的全部或部分步骤可以通过程序指令相关的硬件来完成,前述的程序可以存储于计算机可读取存储介质中,该程序在执行时,执行包括上述方法实施例的步骤;而前述的存储介质包括:移动存储设备、只读存储器(Read Only Memory,ROM)、磁碟或者光盘等各种可以存储程序代码的介质。或者,本申请上述集成的单元如果以软件功能模块的形式实现并作为独立的产品销售或使用时,也可以存储在一个计算机可读取存储介质中。基于这样的理解,本申请实施例的技术方案本质上或者说对相关技术做出贡献的部分可以以软件产品的形式体现出来,该计算机软件产品存储在一个存储介质中,包括若干指令用以使得设备自动测试线执行本申请各个实施例所述方法的全部或部分。而前述的存储介质包括:移动存储设备、ROM、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述,仅为本申请的实施方式,但本申请的保护范围并不局限于此,任何熟悉本技术领域的技术人员在本申请揭露的技术范围内,可轻易想到变化或替换,都应涵盖在本申请的保护范围之内。因此,本申请的保护范围应以所述权利要求的保护范围为准。
工业实用性
本申请实施例中,首先,获取第一地图中的第一图像特征;然后,根据所述第一图像特征,从第二地图匹配出第二图像特征;最后,根据所述第一图像特征和所述第二图像特征,将所述第一地图中每一关键帧图像添加到所述第二地图中,得到更新的第二地图;如此,通过提取局部地图第一地图中的关键帧图像的图像特征,然后将该图像特征,与另一地图中关键帧图像的图像特征进行匹配,即可得到第二图像特征对应的关键帧图像,从而基于不同地图中相匹配的图像特征,将多个地图进行融合,从而实现了地图的更新,提高了更新后的地图的精度高。

Claims (20)

  1. 一种地图更新方法,其中,所述方法包括:
    获取第一地图中的第一图像特征;
    根据所述第一图像特征,从第二地图匹配出第二图像特征;其中,所述第一地图和所述第二地图对应的场景信息部分相同;
    根据所述第一图像特征和所述第二图像特征,将所述第一地图中每一关键帧图像添加到所述第二地图中,得到更新的第二地图。
  2. 根据权利要求1所述的方法,其中,所述获取第一地图中的第一图像特征,包括:
    提取所述第一地图中的待匹配关键帧图像,得到第一图像特征;其中,所述待匹配关键帧图像为所述第一地图中的关键帧图像;
    对应地,所述根据所述第一图像特征,从第二地图匹配出第二图像特征,包括:
    根据所述第一图像特征,从所述第二地图中关键帧图像的图像特征中,匹配出所述第二图像特征。
  3. 根据权利要求2所述的方法,其中,根据所述第一图像特征和所述第二图像特征,将所述第一地图中每一关键帧图像添加到所述第二地图中,得到更新的第二地图,包括:
    获取用于采集所述待匹配关键帧图像的图像采集设备,在所述第一地图所处的第一坐标系中的第一位置信息;
    根据所述第一图像特征和所述第二图像特征,确定所述图像采集设备在所述第二地图所处的第二坐标系中的第二位置信息;
    根据所述第一位置信息和所述第二位置信息,将所述第一地图中每一关键帧图像添加到所述第二地图中,得到更新的第二地图。
  4. 根据权利要求3所述的方法,其中,所述根据所述第一位置信息和所述第二位置信息,将所述第一地图中每一关键帧图像添加到所述第二地图中,得到更新的第二地图,包括:
    根据所述第一位置信息和所述第二位置信息,确定所述第一坐标系和所述第二坐标系之间的转换关系;
    根据所述转换关系,对所述第一地图中的每一关键帧图像对应的图像采集设备在第一坐标系中的坐标进行调整,使得所述每一关键帧图像对应的图像采集设备的调整后的坐标与所述第二坐标系相匹配;
    将每一坐标调整后的图像采集设备对应的关键帧图像,添加到所述第二地图中,得到所述更新的第二地图。
  5. 根据权利要求1所述的方法,其中,所述第一图像特征包括:所述第一地图中的关键帧图像的特征点的二维位置信息、三维位置信息和标识信息;
    所述第二图像特征包括:所述第二地图中的关键帧图像的特征点的二维位置信息和三维位置信息和标识信息;其中,所述三维位置信息是将所述二维位置信息映射在所述二维位置信息所处的坐标系中得到的。
  6. 根据权利要求2所述的方法,其中,所述对第一地图中的关键帧图像进行提取,得到第一图像特征,包括:
    提取所述第一地图中的每一关键帧图像的特征点,得到特征点集合;
    确定所述特征点集合中每一特征点的标识信息和每一所述特征点在所述关键帧图像中的二维位置信息;
    将每一二维位置信息分别映射在所述第一地图所处的坐标系中,得到每一所述特征点的三维位置信息。
  7. 根据权利要求2所述的方法,其中,所述根据所述第一图像特征,从所述第二地图中关键帧图像的图像特征中,匹配出第二图像特征,包括:
    分别确定不同的样本特征点在所述特征点集合中所占的比值,得到第一比值向量;
    获取第二比值向量,所述第二比值向量为所述多个样本特征点在所述第二地图中关键帧图像中包含的特征点中所占的比值;
    根据所述第一图像特征、所述第一比值向量和所述第二比值向量,从所述第二地图中的关键帧 图像的图像特征中,匹配出第二图像特征。
  8. 根据权利要求7所述的方法,其中,所述根据所述第一图像特征、所述第一比值向量和所述第二比值向量,从所述第二地图中的关键帧图像的图像特征中,匹配出第二图像特征,包括:
    根据所述第一比值向量和所述第二比值向量,从所述第二地图中的关键帧图像的图像特征中,确定与所述第一图像特征的相似度大于第二阈值的相似图像特征;
    确定所述相似图像特征所属的相似关键帧图像,得到相似关键帧图像集合;
    从所述相似关键帧图像的图像特征中,选择与所述第一图像特征相似度满足预设相似度阈值的第二图像特征。
  9. 根据权利要求8所述的方法,其中,所述从所述相似关键帧图像的图像特征中,选择与所述第一图像特征相似度满足预设相似度阈值的第二图像特征,包括:
    确定至少两个所述相似关键帧图像的采集时间之间的时间差,和所述至少两个相似关键帧图像的图像特征分别与所述第一图像特征的相似度差;
    将所述时间差小于第三阈值,且所述相似度差小于第四阈值的相似关键帧图像进行联合,得到联合帧图像;
    从所述联合帧图像的图像特征中,选择与所述第一图像特征相似度满足预设相似度阈值的第二图像特征。
  10. 根据权利要求9所述的方法,其中,所述从所述联合帧图像的图像特征中,选择与所述第一图像特征相似度满足预设相似度阈值的第二图像特征,包括:
    分别确定多个联合帧图像中包含的每一关键帧图像的图像特征与所述第一图像特征的相似度之和;
    将相似度之和最大的联合帧图像,确定为与所待匹配关键帧图像的相似度最高的目标联合帧图像;
    根据目标联合帧图像的特征点的标识信息和所述待匹配关键帧图像的特征点的标识信息,从所述目标联合帧图像的图像特征中,选择与所述第一图像特征相似度满足预设相似度阈值的第二图像特征。
  11. 根据权利要求3至10任一项所述的方法,其中,在所述根据所述第一图像特征和所述第二图像特征,确定所述图像采集设备在所述第二地图所处的第二坐标系中的第二位置信息之前,所述方法还包括:
    确定所述第二图像特征对应的关键帧图像中包含的任意两个特征点之间,小于第一阈值的目标欧式距离,得到目标欧式距离集合;
    对应地,所述根据所述第一图像特征和所述第二图像特征,确定所述图像采集设备在所述第二地图所处的第二坐标系中的第二位置信息,包括:
    如果所述目标欧式距离集合中包含的目标欧式距离的数量大于第五阈值,基于所述第二图像特征对应的关键帧图像的特征点的三维位置信息和所述第一图像特征对应的关键帧图像的特征点的二维位置信息,确定所述第二位置信息。
  12. 根据权利要求3至10任一项所述的方法,其中,在所述对第一地图中的关键帧图像进行提取,得到第一图像特征之前,所述方法还包括:
    从样本图像库中,选择满足预设条件的关键帧图像,得到关键帧图像集合;
    提取每一关键帧图像的图像特征,得到关键图像特征集合;
    提取样本图像的特征点,得到包含不同的特征点的样本特征点集合;
    确定每一样本特征点,在所述关键帧图像的中所占的比值,得到比值向量集合;
    存储所述比值向量集合和所述关键图像特征集合,得到所述第一地图。
  13. 根据权利要求12所述的方法,其中,在所述从样本图像库中,选择满足预设条件的关键帧图像,得到关键帧图像集合之前,所述方法还包括:
    从所述样本图像中选择预设数量的角点;
    如果采集时间相邻的两个样本图像中包含的相同的角点数量大于等于第六阈值,确定所述样本图像对应的场景为连续场景;
    如果采集时间相邻的两个样本图像中包含的相同的角点数量小于第六阈值,确定所述样本图像对应的场景为离散场景。
  14. 根据权利要求12所述的方法,其中,从样本图像库中,选择满足预设条件的关键帧图像,得到关键帧图像集合,包括:
    如果所述样本图像对应的场景为离散场景,根据输入的选择指令,从样本图像库中选择关键帧图像;
    如果所述样本图像对应的场景为连续场景,根据预设的帧率或视差,从样本图像库中选择关键帧图像。
  15. 根据权利要求12所述的方法,其中,所述确定每一样本特征点,在所述关键帧图像的中所占的比值,得到比值向量集合,包括:
    根据样本图像库中包含的样本图像的第一数量和第i个样本特征点在样本图像库中出现的第一次数,确定第一平均次数;其中,i为大于等于1的整数;所述第一平均次数用于表明所述第i个样本特征点平均在每一样本图像中出现的次数;
    根据所述第i个样本特征点在第j个关键帧图像中出现的第二次数和所述第j个关键帧图像中包含的样本特征点的第二数量,确定第二平均次数;其中,j为大于等于1的整数;所述第二平均次数用于表明所述第i个样本特征点占据第j个关键帧图像中包含的样本特征点的比例;
    根据所述第一平均次数和所述第二平均次数,得到样本特征点在所述关键帧图像的中所占的比值,得到所述比值向量集合。
  16. 一种地图更新装置,其中,所述装置包括:第一获取模块、第一匹配模块和第一更新模块,其中:
    所述第一获取模块,配置为获取第一地图中的第一图像特征;
    所述第一匹配模块,配置为根据所述第一图像特征,从第二地图匹配出第二图像特征;
    所述第一更新模块,配置为根据所述第一图像特征和所述第二图像特征,将所述第一地图中每一关键帧图像添加到所述第二地图中,得到更新的第二地图。
  17. 根据权利要求16所述的装置,其中,所述第一获取模块,包括:
    第一提取子模块,配置为提取所述第一地图中的待匹配关键帧图像,得到第一图像特征;其中,所述待匹配关键帧图像为所述第一地图中的关键帧图像;
    对应地,所述第一匹配模块,包括:
    第一匹配子模块,配置为根据所述第一图像特征,从所述第二地图中关键帧图像的图像特征中,匹配出所述第二图像特征。
  18. 根据权利要求16所述的装置,其中,所述第一更新模块,包括:
    第一获取子模块,配置为获取用于采集所述待匹配关键帧图像的图像采集设备在所述第一地图所处的第一坐标系中的第一位置信息;
    第一确定子模块,配置为根据所述第一图像特征和所述第二图像特征,确定所述图像采集设备在所述第二地图所处的第二坐标系中的第二位置信息;
    第一更新子模块,配置为根据所述第一位置信息和所述第二位置信息,将所述第一地图中每一关键帧图像添加到所述第二地图中,得到更新的第二地图。
  19. 一种终端,包括存储器和处理器,所述存储器存储有可在处理器上运行的计算机程序,其中,所述处理器执行所述程序时实现权利要求1至15任一项所述方法中的步骤。
  20. 一种计算机可读存储介质,其上存储有计算机程序,其中,该计算机程序被处理器执行时实现权利要求1至15任一项所述方法中的步骤。
PCT/CN2020/096505 2019-06-28 2020-06-17 地图更新方法及装置、终端、存储介质 WO2020259361A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP20831138.1A EP3975003A4 (en) 2019-06-28 2020-06-17 MAP UPDATING METHOD AND APPARATUS AND TERMINAL AND STORAGE MEDIA
US17/549,840 US20220099455A1 (en) 2019-06-28 2021-12-13 Map update method, terminal and storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910578745.9A CN112148742A (zh) 2019-06-28 2019-06-28 地图更新方法及装置、终端、存储介质
CN201910578745.9 2019-06-28

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/549,840 Continuation US20220099455A1 (en) 2019-06-28 2021-12-13 Map update method, terminal and storage medium

Publications (1)

Publication Number Publication Date
WO2020259361A1 true WO2020259361A1 (zh) 2020-12-30

Family

ID=73891185

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/096505 WO2020259361A1 (zh) 2019-06-28 2020-06-17 地图更新方法及装置、终端、存储介质

Country Status (4)

Country Link
US (1) US20220099455A1 (zh)
EP (1) EP3975003A4 (zh)
CN (1) CN112148742A (zh)
WO (1) WO2020259361A1 (zh)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210404834A1 (en) * 2020-06-30 2021-12-30 Lyft, Inc. Localization Based on Multi-Collect Fusion
CN112749350B (zh) * 2021-01-14 2024-05-14 Oppo广东移动通信有限公司 推荐对象的信息处理方法、装置、存储介质与电子设备
CN113706592A (zh) * 2021-08-24 2021-11-26 北京百度网讯科技有限公司 一种定位信息的修正方法、装置、电子设备和存储介质
CN117459688B (zh) * 2023-12-26 2024-05-03 海纳云物联科技有限公司 基于地图系统的摄像头角度标示方法、装置及介质

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101275854A (zh) * 2007-03-26 2008-10-01 日电(中国)有限公司 更新地图数据的方法和设备
CN102289809A (zh) * 2011-07-25 2011-12-21 清华大学 估计摄像机位姿的方法及装置
CN103123727A (zh) * 2011-11-21 2013-05-29 联想(北京)有限公司 即时定位与地图构建方法和设备
CN105447911A (zh) * 2014-09-26 2016-03-30 联想(北京)有限公司 一种3d地图融合方法、装置及电子设备
CN107167149A (zh) * 2017-06-26 2017-09-15 上海与德科技有限公司 一种街景视图制作方法及系统
CN109029422A (zh) * 2018-07-10 2018-12-18 北京木业邦科技有限公司 一种多无人机协作构建三维调查地图的方法和装置
CN109074757A (zh) * 2018-07-03 2018-12-21 深圳前海达闼云端智能科技有限公司 一种建立地图的方法、终端和计算机可读存储介质
CN109086277A (zh) * 2017-06-13 2018-12-25 纵目科技(上海)股份有限公司 一种重叠区构建地图方法、系统、移动终端及存储介质
CN109285117A (zh) * 2018-09-05 2019-01-29 南京理工大学 一种基于地图特征的多地图拼接融合算法

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10185775B2 (en) * 2014-12-19 2019-01-22 Qualcomm Technologies, Inc. Scalable 3D mapping system
JP6673276B2 (ja) * 2017-03-28 2020-03-25 カシオ計算機株式会社 音声検出装置、音声検出方法、及びプログラム
CN107301402B (zh) * 2017-06-30 2020-06-16 锐捷网络股份有限公司 一种现实场景关键帧的确定方法、装置、介质及设备
US10482674B1 (en) * 2018-06-27 2019-11-19 Beijing Jingdong Shangke Information Technology Co., Ltd. System and method for mobile augmented reality
JP7139762B2 (ja) * 2018-07-31 2022-09-21 カシオ計算機株式会社 自律移動装置、自律移動方法及びプログラム
CN109544615B (zh) * 2018-11-23 2021-08-24 深圳市腾讯信息技术有限公司 基于图像的重定位方法、装置、终端及存储介质
EP3939006A1 (en) * 2019-03-15 2022-01-19 RetinAI Medical AG Feature point detection

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101275854A (zh) * 2007-03-26 2008-10-01 日电(中国)有限公司 更新地图数据的方法和设备
CN102289809A (zh) * 2011-07-25 2011-12-21 清华大学 估计摄像机位姿的方法及装置
CN103123727A (zh) * 2011-11-21 2013-05-29 联想(北京)有限公司 即时定位与地图构建方法和设备
CN105447911A (zh) * 2014-09-26 2016-03-30 联想(北京)有限公司 一种3d地图融合方法、装置及电子设备
CN109086277A (zh) * 2017-06-13 2018-12-25 纵目科技(上海)股份有限公司 一种重叠区构建地图方法、系统、移动终端及存储介质
CN107167149A (zh) * 2017-06-26 2017-09-15 上海与德科技有限公司 一种街景视图制作方法及系统
CN109074757A (zh) * 2018-07-03 2018-12-21 深圳前海达闼云端智能科技有限公司 一种建立地图的方法、终端和计算机可读存储介质
CN109029422A (zh) * 2018-07-10 2018-12-18 北京木业邦科技有限公司 一种多无人机协作构建三维调查地图的方法和装置
CN109285117A (zh) * 2018-09-05 2019-01-29 南京理工大学 一种基于地图特征的多地图拼接融合算法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3975003A4 *

Also Published As

Publication number Publication date
US20220099455A1 (en) 2022-03-31
EP3975003A1 (en) 2022-03-30
CN112148742A (zh) 2020-12-29
EP3975003A4 (en) 2022-07-27

Similar Documents

Publication Publication Date Title
WO2020259361A1 (zh) 地图更新方法及装置、终端、存储介质
WO2021057797A1 (zh) 定位方法及装置、终端、存储介质
WO2020259360A1 (zh) 定位方法及装置、终端、存储介质
WO2021057744A1 (zh) 定位方法及装置、设备、存储介质
CN110568447B (zh) 视觉定位的方法、装置及计算机可读介质
WO2021057742A1 (zh) 定位方法及装置、设备、存储介质
Ventura et al. Global localization from monocular slam on a mobile phone
US20090083275A1 (en) Method, Apparatus and Computer Program Product for Performing a Visual Search Using Grid-Based Feature Organization
EP3274964B1 (en) Automatic connection of images using visual features
CN105023266A (zh) 增强现实实现方法、装置和终端设备
CN110738703B (zh) 定位方法及装置、终端、存储介质
US9288636B2 (en) Feature selection for image based location determination
CN111024089A (zh) 一种基于bim与计算机视觉技术的室内定位导航方法
JP2023510945A (ja) シーン識別方法及びその装置、インテリジェントデバイス、記憶媒体並びにコンピュータプログラム
US9811539B2 (en) Hierarchical spatial clustering of photographs
Zhang et al. Research on 3D architectural scenes construction technology based on augmented reality
KR100327120B1 (ko) 증강현실 기반의 3차원 트래커와 컴퓨터 비젼을 이용한영상 정합 방법
US9188444B2 (en) 3D object positioning in street view
US9852542B1 (en) Methods and apparatus related to georeferenced pose of 3D models
CN105677843A (zh) 一种自动获取宗地四至属性的方法
EP3300020A1 (en) Image based location determination
Li et al. A sparse feature matching model using a transformer towards large-view indoor visual localization
US20150134689A1 (en) Image based location determination
Wang et al. ReLoc: indoor visual localization with hierarchical sitemap and view synthesis
US20150379040A1 (en) Generating automated tours of geographic-location related features

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20831138

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2020831138

Country of ref document: EP

Effective date: 20211223