WO2022142049A1 - 地图构建方法及装置、设备、存储介质、计算机程序产品 - Google Patents

地图构建方法及装置、设备、存储介质、计算机程序产品 Download PDF

Info

Publication number
WO2022142049A1
WO2022142049A1 PCT/CN2021/092114 CN2021092114W WO2022142049A1 WO 2022142049 A1 WO2022142049 A1 WO 2022142049A1 CN 2021092114 W CN2021092114 W CN 2021092114W WO 2022142049 A1 WO2022142049 A1 WO 2022142049A1
Authority
WO
WIPO (PCT)
Prior art keywords
key frame
image
map
frame image
newly registered
Prior art date
Application number
PCT/CN2021/092114
Other languages
English (en)
French (fr)
Inventor
周立阳
孙瀚
冯友计
姜翰青
章国锋
Original Assignee
浙江商汤科技开发有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 浙江商汤科技开发有限公司 filed Critical 浙江商汤科技开发有限公司
Publication of WO2022142049A1 publication Critical patent/WO2022142049A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/22—Matching criteria, e.g. proximity measures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/05—Geographic models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30244—Camera pose

Definitions

  • the present disclosure relates to, but is not limited to, the field of computer vision, and in particular, to a map construction method and apparatus, device, storage medium, computer program product, and computer program.
  • 3D reconstruction technology has broad application prospects in the fields of cultural relics protection, positioning and navigation, scene roaming, augmented reality, and industrial measurement.
  • Three-dimensional reconstruction methods can be divided into two categories: incremental reconstruction methods and global reconstruction methods.
  • Embodiments of the present disclosure provide a map construction method and apparatus, device, storage medium, computer program product, and computer program, which can be implemented as follows:
  • An embodiment of the present disclosure provides a map construction method, the method is executed by an electronic device, and the method includes: determining whether the newly registered image belongs to a key frame according to an association feature between a newly registered image and an existing key frame image image; each time a new map point is added to the constructed three-dimensional map based on the newly determined key frame image.
  • the determining whether the newly registered image belongs to the key frame image according to the association feature between the newly registered image and the existing key frame image includes: determining whether the newly registered image belongs to the existing key frame image The loopback frame corresponding to the key frame image; if the newly registered image belongs to the loopback frame corresponding to the existing key frame image, it is determined that the new registered image belongs to the key frame image.
  • the determining whether the newly registered image belongs to the key frame image according to the association feature between the newly registered image and the existing key frame image includes: when the newly registered image does not belong to the existing key frame image In the case of the loopback frame corresponding to the key frame image, determine the total number of target feature points included in the newly registered image; wherein, the target feature points are those with corresponding map points on the constructed three-dimensional map. feature points; if the total number of the target feature points is less than a preset number, it is determined that the newly registered image belongs to the key frame image.
  • the determining whether the new registered image belongs to the key frame image according to the association feature between the new registered image and the existing key frame image includes: respectively determining that the camera is collecting the new registered image and the first When a key frame image is used, the first coordinate and the second coordinate in the world coordinate system; wherein, the first key frame image is the key frame whose acquisition time is closest to the newly registered image among the existing key frame images image; determining a first distance value between the first coordinate and the second coordinate; determining that the newly registered image belongs to the key frame image if the first distance value satisfies the preset distance requirement .
  • the preset distance requirement includes satisfying the corresponding relationship with the target distance value; the method further includes: in the existing key frame images, respectively determining two adjacent camera acquisition time When a key frame image is taken, the third coordinate and the fourth coordinate in the world coordinate system; the second distance value between the third coordinate and the fourth coordinate is determined; the second distance value is determined according to the second distance value. target distance value.
  • the determining whether the newly registered image belongs to the key frame image according to the association feature between the newly registered image and the existing key frame image includes: determining the newly registered image and the first key frame image
  • the first key frame image is the key frame image whose acquisition time is closest to the newly registered image among the existing key frame images; when the image parallax is greater than the preset parallax , and determine that the newly registered image belongs to the key frame image.
  • the step of adding a new map point to the constructed three-dimensional map based on the newly determined key frame image each time includes: performing a step on the feature points included in the newly determined key frame image. Sampling to obtain multiple sampling points; on the constructed three-dimensional map, new map points are generated according to the multiple sampling points.
  • the sampling of the newly determined feature points included in the key frame image to obtain a plurality of sampling points includes: among the newly determined feature points included in the key frame image, determining A second feature point that does not match the map point on the constructed three-dimensional map; on the newly determined key frame image, sampling the second feature point to obtain the plurality of sampling points.
  • sampling the second feature point on the newly determined key frame image to obtain the plurality of sampling points includes: on the newly determined key frame image, According to the preset sampling radius, a plurality of sampling regions are divided; the second feature points included in each sampling region are uniformly sampled to obtain the plurality of sampling points.
  • performing uniform sampling on the second feature points included in each sampling area to obtain the plurality of sampling points includes: sampling the same number of the second feature points in each sampling area For the second feature point, obtain the plurality of sampling points whose total number satisfies the preset interval range.
  • the method further includes: optimizing the constructed three-dimensional map according to the at least one second key frame image.
  • the method further includes: using the key frame image newly determined each time as the second key frame image; or using multiple key frame images determined in multiple times as multiple first key frame images Two keyframe images.
  • the optimizing the constructed three-dimensional map according to at least one second key frame image includes: determining that at least one map point on the constructed three-dimensional map is in each second key frame The corresponding weight value in the image; based on the corresponding weight value of the at least one map point in each second key frame image, determine the loss function corresponding to the constructed three-dimensional map; determine the camera that minimizes the loss function Pose parameters; based on the camera pose parameters, update the at least one map point on the constructed three-dimensional map.
  • the determining a weight value corresponding to at least one map point on the constructed three-dimensional map in each second key frame image includes: at least one map point on the constructed three-dimensional map. Among the map points, determine a first map point that does not have a corresponding feature point and a second map point that has a corresponding feature point in the current second key frame image; determine that the first map point is in the current second key frame image. The corresponding weight value in the frame image is 0, and it is determined that the weight value corresponding to the second map point in the current second key frame image is proportional to the target value.
  • the method further includes: in the existing key frame images, determining the total number of third key frame images; wherein the second map point exists on the third key frame images corresponding feature points; based on the total number of the third key frame images, determine the target value corresponding to the second map point.
  • the method further includes: determining the feature points included in each key frame image in the existing key frame images; using any two key frame images in the existing key frame images as a group image pairs, match the feature points of any group of image pairs, and determine the matching result; according to the matching results, after sorting the multiple groups of image pairs in a pre-specified order, the order is based on different image pairs.
  • Reconstruction initialization build a three-dimensional map based on the initialization image pairs that are successfully reconstructed and initialized in the multiple groups of image pairs.
  • An embodiment of the present disclosure provides an apparatus for constructing a map, the apparatus includes: a first determination part configured to determine whether the newly registered image belongs to a key frame image according to an association feature between a newly registered image and an existing key frame image ;
  • the map update part is configured to add new map points on the constructed three-dimensional map each time based on the newly determined key frame image.
  • Embodiments of the present disclosure provide a computer-readable storage medium, where the storage medium stores a computer program, and the computer program is configured to execute part or all of the steps of the map construction method described in the embodiments of the present disclosure.
  • An embodiment of the present disclosure provides a map construction device, including: a processor; a memory configured to store executable instructions of the processor; wherein the processor is configured to call the executable instructions stored in the memory to achieve Part or all of the steps of the map construction method described in the embodiments of the present disclosure.
  • An embodiment of the present disclosure provides a computer program product, the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, the Some or all of the steps of the described map construction method.
  • the embodiments of the present disclosure provide a computer program, the computer program includes computer-readable codes, and when the computer-readable codes are read and executed by a computer, the method for implementing the map construction method described in the embodiments of the present disclosure is realized. some or all of the steps.
  • a key frame image may be selected from the newly registered image according to the association feature between the newly registered image and the existing key frame image, and each time based on the newly determined key frame image, a key frame image may be displayed on the constructed three-dimensional map. Add new map points.
  • the embodiment of the present disclosure reduces the redundancy of image data in the process of incremental reconstruction of the three-dimensional map, simplifies the reconstruction scale, and improves the reconstruction efficiency.
  • the newly registered image belongs to the loopback frame corresponding to the existing key frame image
  • the newly registered image belongs to the key frame image.
  • the loopback frame corresponding to the existing key frame image is regarded as a relatively important key frame image, which improves the accuracy of map reconstruction and facilitates the protection of the map reconstruction scene.
  • the total number of target feature points included in the newly registered image can be determined on the constructed three-dimensional map.
  • the target feature point is a feature point that has a corresponding map point on the constructed three-dimensional map.
  • the first coordinate and the second coordinate in the world coordinate system when the camera collects the newly registered image and the first key frame image can be respectively determined, wherein the first key frame image is an existing key frame image A keyframe image that is closest to the acquisition time of the newly registered image.
  • a first distance value between the first coordinate and the second coordinate may be determined, and if the first distance value meets a preset distance requirement, it may be determined that the newly registered image belongs to the keyframe image.
  • whether the newly registered image belongs to the key frame image may be determined according to the position in the world coordinate system when the camera collects the newly registered image and the first key frame image. The redundancy of image data during the incremental reconstruction of the three-dimensional map is reduced, the reconstruction scale is simplified, and the reconstruction efficiency is improved.
  • the third and fourth coordinates in the world coordinate system can be determined according to the existing key frame images when the camera collects two adjacent key frame images, so as to determine the third and fourth coordinates The second distance value between.
  • the target distance value is determined according to the second distance value.
  • the embodiment of the present disclosure can control the insertion frequency of key frame images by adjusting the relationship between the first distance value and the target distance value, reduce the redundancy of image data in the process of incremental reconstruction of the 3D map, simplify the reconstruction scale, and improve the reconstruction efficiency .
  • whether the newly registered image belongs to the key frame image may also be determined according to the image parallax between the newly registered image and the first key frame image whose acquisition time is closest among the existing key frame images. Make full use of the temporal structure of the input registration image, and determine the key frame image under the condition of ensuring the connectivity of the scene. Each time, a new map point can be added to the constructed 3D map according to the newly determined key frame image. The redundancy of image data during the incremental reconstruction of the three-dimensional map is reduced, the reconstruction scale is simplified, and the reconstruction efficiency is improved.
  • the feature points included in the newly determined key frame image may be sampled to obtain multiple sampling points.
  • new map points are generated according to multiple sampling points.
  • uniform sampling is performed for the second feature point that has no corresponding map point on the constructed three-dimensional map, and by adjusting the sampling radius, the sampling point can be controlled. total number. In this way, it is possible to reduce the over-concentration of image features, reduce the complexity of subsequent optimization, and improve reconstruction efficiency.
  • the constructed three-dimensional map may be optimized according to at least one second key frame image.
  • the key frame image newly determined each time may be used as the second key frame image, so as to perform local optimization on the constructed three-dimensional map after each new determination of the key frame image.
  • multiple key frame images determined multiple times may also be used as multiple second key frame images to perform global optimization on the constructed three-dimensional map. The optimization complexity is reduced and the reconstruction accuracy is improved.
  • At least one map point on the constructed three-dimensional map can be determined, and whether there is a corresponding feature point in each second key frame image.
  • the weight value corresponding to the first map point in the current second key frame image is 0, and it is determined that there is a corresponding feature point.
  • the corresponding weight value in the current second key frame image is proportional to the target value.
  • the total number of third key frame images may be determined in the existing key frame images, and the target value may be determined according to the total number of third key frame images.
  • FIG. 1 is a schematic diagram of an implementation flow of a map construction method provided by an embodiment of the present disclosure
  • FIG. 2 is a schematic diagram of an implementation flow of a map construction method provided by an embodiment of the present disclosure
  • FIG. 3 is a schematic diagram of an implementation flow of a map construction method provided by an embodiment of the present disclosure
  • FIG. 4 is a schematic diagram of an implementation flow of a map construction method provided by an embodiment of the present disclosure
  • FIG. 5 is a schematic diagram of an implementation flow of a map construction method provided by an embodiment of the present disclosure
  • FIG. 6 is a schematic diagram of an implementation flow of a map construction method provided by an embodiment of the present disclosure.
  • FIG. 7 is a schematic diagram of an implementation flow of a map construction method provided by an embodiment of the present disclosure.
  • FIG. 8 is a schematic diagram of a scenario of uniformly sampling second feature points according to an embodiment of the present disclosure.
  • FIG. 9 is a schematic diagram of an implementation flow of a map construction method provided by an embodiment of the present disclosure.
  • FIG. 10 is a schematic diagram of an implementation flow of a map construction method provided by an embodiment of the present disclosure.
  • FIG. 11 is a schematic diagram of an implementation flow of a map construction method provided by an embodiment of the present disclosure.
  • FIG. 12 is a schematic diagram of an implementation flow of a map construction method provided by an embodiment of the present disclosure.
  • FIG. 13 is a schematic diagram of an implementation flowchart of a map construction method provided by an embodiment of the present disclosure
  • 14A is a schematic diagram of a scenario for determining a baseline provided by an embodiment of the present disclosure.
  • FIG. 14B is a schematic diagram of an implementation flow of a map construction method provided by an embodiment of the present disclosure.
  • FIG. 15 is a schematic diagram of the composition and structure of a map construction apparatus provided by an embodiment of the present disclosure.
  • FIG. 16 is a schematic structural diagram of a map construction device provided by an embodiment of the present disclosure.
  • first, second, third, etc. may be used in this disclosure to describe various pieces of information, such information should not be limited by these terms. These terms are only used to distinguish the same type of information from each other.
  • first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information, without departing from the scope of the present disclosure.
  • word “if” as used herein may be interpreted as "at the time of” or "when” or "in response to determining.”
  • the incremental reconstruction method can achieve higher reconstruction accuracy when reconstructing a small-scale scene.
  • the reconstruction efficiency drops sharply, and at the same time, because of the frame-by-frame registration method of the incremental reconstruction process, the large scene reconstruction will cause drift.
  • the global reconstruction can directly estimate the camera pose through the scene graph, and only needs to perform a global adjustment process, which is suitable for the task of large-scale scene reconstruction.
  • the global reconstruction is sensitive to matching noise and is not fully optimized, so the robustness and accuracy of the reconstruction are degraded.
  • FIG. 1 is a schematic diagram of an implementation flow of a map construction method provided by an embodiment of the present disclosure.
  • the method may be executed by an electronic device, and the electronic device may be a terminal device or a server or other processing device, wherein the terminal device It can be User Equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, Personal Digital Assistant (PDA), handheld device, computing device, vehicle-mounted device, wearable device, etc.
  • the server can be a physical server, a virtual machine, a cloud server, a server cluster, and the like.
  • the method may be implemented by a processor invoking computer-readable instructions stored in a memory. As shown in Figure 1, the method may include the following steps:
  • step 101 it is determined whether the newly registered image belongs to the key frame image according to the association feature between the newly registered image and the existing key frame image.
  • the newly registered image is an image newly acquired by the camera each time during the moving process.
  • the key frame images are relatively important images screened from the newly registered images according to the time sequence from front to back.
  • 3D map reconstruction can be performed based on the key frame images.
  • step 102 new map points are added to the constructed three-dimensional map each time based on the newly determined key frame image.
  • the constructed three-dimensional map is a three-dimensional map constructed after performing three-dimensional reconstruction and initialization based on an existing key frame image. After each new key frame image is determined, according to the feature points included in the newly determined key frame image, a new three-dimensional map point is added to the constructed three-dimensional map by the method of three-dimensional point matching, so as to realize the reconstruction of the three-dimensional map. .
  • the key frame image can be selected from the newly registered image according to the association feature between the newly registered image and the existing key frame image, and each time based on the newly determined key frame image, an image is added to the constructed three-dimensional map. New map points.
  • the embodiment of the present disclosure reduces the redundancy of image data in the process of incremental reconstruction of the three-dimensional map, simplifies the reconstruction scale, and improves the reconstruction efficiency.
  • At least one of the following manners may be used to determine whether the newly registered image belongs to a key frame image.
  • whether the newly registered image belongs to the key frame image is determined according to whether the newly registered image belongs to the loopback frame corresponding to the existing key frame image.
  • step 101 may include:
  • step 101-11 it is determined whether the newly registered image belongs to the loopback frame corresponding to the existing key frame image.
  • the loopback frame refers to an image obtained by shooting a scene again after a certain period of time after shooting the scene.
  • the loopback frame is to ensure the accuracy of scene reconstruction. Therefore, loop closure frames are important in the 3D reconstruction process.
  • the newly registered image is not based on the degree of matching of the image features between the newly registered image and the existing key frame image, and the number of registered images spaced between the new registered image and the existing key frame image. It belongs to the loopback frame corresponding to the existing keyframe image.
  • the matching degree of the image features between the newly registered image and the existing key frame image exceeds the preset matching degree, and the number of registered images in the interval exceeds the preset number of images, it can be considered that the newly registered image belongs to the existing key frame
  • the loopback frame corresponding to the image
  • steps 101-12 in the case that the newly registered image belongs to the loopback frame corresponding to the existing key frame image, it is determined that the newly registered image belongs to the key frame image.
  • the loop closure frame is directly used as the key frame image, and the three-dimensional map reconstruction can be performed subsequently according to the key frame image.
  • the loopback frame is regarded as a relatively important key frame image, which improves the accuracy of map reconstruction and facilitates the protection of the map reconstruction scene.
  • step 101 may include:
  • steps 101-21 if the newly registered image does not belong to the loop closure frame corresponding to the existing key frame image, the total number of target feature points included in the newly registered image is determined.
  • a three-dimensional point matching method may be used to determine that each feature point included in the newly registered image is in the Whether there is a corresponding map point on the constructed three-dimensional map, if there is a corresponding map point, the feature point is the target feature point.
  • the embodiment of the present disclosure can determine the total number of target feature points included in the newly registered image.
  • steps 101-22 when the total number of the target feature points is less than a preset number, it is determined that the newly registered image belongs to the key frame image.
  • the preset number may be a positive integer greater than zero, such as 300.
  • the total number of target feature points included in the newly registered image can be determined on the constructed three-dimensional map.
  • the target feature point is a feature point that has a corresponding map point on the constructed three-dimensional map.
  • step 101 may include:
  • steps 101-31 respectively determine the first coordinate and the second coordinate in the world coordinate system when the camera acquires the new registration image and the first key frame image.
  • the position of the camera in the world coordinate system should not move very far. Therefore, the first coordinates and the second coordinates in the world coordinate system of the camera when the new registration image and the first key frame image are acquired can be determined respectively.
  • the first key frame image is a key frame image whose acquisition time is closest to the newly registered image among the existing key frame images, that is, the first key frame image is a key frame image whose acquisition time is closest to the newly registered image.
  • the first coordinate and the second coordinate can be represented by the pose of the camera in the world coordinate system.
  • random sample consistency Random Sample Consensus, RANSAC
  • Perspective-n-Points Perspective-n-Points
  • P3P P3P algorithm estimates the pose of the camera in the world coordinate system when the new registration image is collected, so as to determine the first coordinate.
  • the second coordinate in the world coordinate system can also be determined when the camera captures the first key frame image.
  • a first distance value between the first coordinate and the second coordinate is determined.
  • the first distance value between the two may be calculated in the world coordinate system.
  • step 101-33 in the case that the first distance value meets the preset distance requirement, it is determined that the newly registered image belongs to the key frame image.
  • the preset distance requirement may include, but is not limited to, satisfying a corresponding relationship with the target distance value, and the corresponding relationship may include, but is not limited to, the product relationship between the target distance value and a preset value greater than 0.
  • the average value of the second distance values between the four coordinates is used as the target distance value.
  • the target distance value It can be represented by the following formula 1-1:
  • C k represents the coordinates of the camera in the world coordinate system when the k-th key frame image is collected
  • f(k) represents the key frame image index closest to the k-th key frame image acquisition time
  • K is the existing key
  • the total number of frame images, k is an integer greater than 1.
  • C k+1 represents the first coordinate in the world coordinate system when the camera collects a new registered image
  • f(k+1) represents the index of the first key frame image
  • the first key frame image is the existing key frame
  • C f(k+1) represents the second coordinate in the world coordinate system when the camera collects the first key frame image
  • ⁇ is a preset multiple greater than 0.
  • the insertion frequency of key frame images can be controlled by adjusting ⁇ .
  • ⁇ When ⁇ ⁇ 1, formula 1-2 becomes a monotonically increasing function, and the interval between key frame images will become larger and larger.
  • ⁇ the insertion frequency of key frame images can be kept in a fixed range.
  • ⁇ may be located in the interval range of [0.8, 1].
  • the third method can be implemented independently, or the newly registered image can be determined according to the camera moving distance value on the basis of determining that the newly registered image does not belong to the loopback frame corresponding to the existing key frame image in the above steps 101-11. Whether the image is a keyframe image.
  • the insertion frequency of key frame images can be controlled by adjusting the preset multiple, which is beneficial to reduce the redundancy of image data in the process of incremental reconstruction of the 3D map, simplify the reconstruction scale, and improve the reconstruction efficiency.
  • step 101 may include:
  • steps 101-41 the image parallax between the newly registered image and the first key frame image is determined.
  • the first key frame image is the key frame image whose acquisition time is closest to the newly registered image among the existing key frame images.
  • Image parallax refers to the directional difference produced by observing the same target from two different points, which can generally be measured by the parallax angle, that is, the corresponding angle determined when viewing two different points from the same target.
  • a map point may be selected from the constructed three-dimensional map, and the map point has a corresponding feature point on both the newly registered image and the first key frame image.
  • steps 101-42 in the case that the image parallax is greater than a preset parallax, it is determined that the newly registered image belongs to the key frame image.
  • the preset disparity may be determined according to verification, and may be, for example, a value greater than or equal to zero.
  • the fourth method can be implemented independently, or in the above steps 101-11, on the basis that it is determined that the newly registered image does not belong to the loopback frame corresponding to the existing key frame image, it can be determined according to the image parallax whether the newly registered image is not. is a keyframe image.
  • the camera moving distance value satisfies the preset distance requirement, and the image parallax with the first key frame image is greater than the preset parallax In the case of , it is determined that the newly registered image belongs to the key frame image.
  • step 102 may include:
  • step 102-1 sampling the newly determined feature points included in the key frame image to obtain a plurality of sampling points.
  • uniform sampling may be performed on the feature points included in each newly determined key frame image, so as to reduce the situation that the image features are too concentrated, and obtain multiple sampling points.
  • step 102-2 on the constructed three-dimensional map, a new map point is generated according to the plurality of sampling points.
  • a new map point is generated on the constructed three-dimensional map by matching multiple sampling points with three-dimensional points.
  • step 102-1 may include:
  • step 201 among the newly determined feature points included in the key frame image, a second feature point that does not match the map points on the constructed three-dimensional map is determined.
  • the newly determined key frame image includes a plurality of feature points, and it can be determined whether each feature point has a matching map point on the constructed map, and if there is a matching map point, the The feature point is taken as the first feature point, otherwise the feature point is taken as the second feature point.
  • the existing key frame images include image 1 and image 2, wherein the first feature points 82 in images 1 and 2 have matching map points 81 in the existing three-dimensional map.
  • the newly determined key frame image is image 3.
  • the feature point corresponds to the first feature point 82 .
  • the second feature point 83 corresponds to the second feature point 83 .
  • step 202 the second feature point is sampled on the newly determined key frame image to obtain the plurality of sample points.
  • a second feature point that does not match the map point on the constructed three-dimensional map may be determined, and subsequent sampling is performed based on the second feature point, so as to generate
  • the new map points reduce the generation of map points that duplicate the existing map points, and have high usability.
  • steps 102-12 may include:
  • step 301 on the newly determined key frame image, a plurality of sampling regions are divided according to a preset sampling radius.
  • the preset sampling radius may be divided according to the needs of the total number of sampling points. When the total number is large, the preset sampling radius can be reduced. On the contrary, the preset sampling radius can be increased, so that multiple sampling areas can be divided on the newly determined key frame image. For example, as shown in Figure 8, in image 3 can be divided into sampling area 1, sampling area 2 and sampling area 3.
  • step 302 the second feature points included in each sampling area are uniformly sampled to obtain the plurality of sampling points.
  • the second feature points may be uniformly sampled in each sampling area, and the same number of the second feature points may be sampled to obtain the plurality of sampling points whose total number satisfies the preset interval range.
  • the preset area range can be any integer between [900, 2000].
  • sampling regions 1 and 2 For example, as shown in FIG. 8 , assuming that one second feature point is sampled in each sampling region, only one second feature point is retained in sampling regions 1 and 2 .
  • the second feature point that is closer to the center of the sampling area may be retained. Since there is only one second feature point in the sampling area 3, the second feature point is retained.
  • uniform sampling can be performed for the second feature points that have no corresponding map points on the constructed three-dimensional map, and the total number of sampling points can be controlled by adjusting the sampling radius. It reduces the over-concentration of image features, reduces the complexity of subsequent optimization, and improves reconstruction efficiency.
  • the above method may further include:
  • step 103 the constructed three-dimensional map is optimized according to at least one second key frame image.
  • each time a newly determined key frame image may be used as a second key frame image, and each time a new key frame image is determined, local optimization of the three-dimensional map is performed.
  • the three-dimensional map can also be globally optimized by using the plurality of key frame images determined in multiple times as the plurality of second key frame images.
  • step 103 may include:
  • step 103-1 a weight value corresponding to at least one map point on the constructed three-dimensional map in each second key frame image is determined.
  • the weight value corresponding to each map point in each second key frame image may be determined according to whether each map point has a corresponding feature point on each second key frame image.
  • step 103-2 a loss function corresponding to the constructed three-dimensional map is determined based on the corresponding weight value of the at least one map point in each of the second key frame images.
  • the loss function f(*) can be represented by the following formulas 1-3:
  • n is the total number of existing key frame images
  • m is the total number of map points included in the constructed 3D map
  • K is the camera internal parameter, generally pre-calibrated
  • R i and T i are the camera
  • the corresponding rotation matrix and translation vector when the i-th key frame image is collected can generally be determined by the camera pose.
  • X j is the jth map point on the constructed 3D map.
  • x j is the feature point corresponding to the jth map point on the key frame image.
  • v ij is the weight value of the feature point corresponding to the j th map point on the ith key frame image.
  • step 103-3 the camera pose parameters that minimize the loss function are determined.
  • the purpose of the above local or global optimization is to reduce the value of the function f(*).
  • the camera pose parameters include but are not limited to Ri and T i in the above formulas 1-3 .
  • step 103-4 the at least one map point on the constructed three-dimensional map is updated based on the camera pose parameter.
  • the camera pose parameters corresponding to each map point on the constructed three-dimensional map can be re-determined, so that the map points on the constructed three-dimensional map are updated based on the re-determined camera pose parameters, Realize global optimization or local optimization of the constructed 3D map.
  • step 103-1 may include:
  • step 401 among at least one map point on the constructed three-dimensional map, determine a first map point that does not have a corresponding feature point and a second map point that has a corresponding feature point in the current second key frame image map point.
  • at least one map point on the constructed three-dimensional map needs to be determined in each second key frame image whether there is a corresponding feature point. Thereby, a first map point that does not have a corresponding feature point on the current second key frame image, and a second map point that has a corresponding feature point are obtained.
  • the number of the second key frame image is 1, it is necessary to determine whether the 5 map points have corresponding feature points on the second key frame image. In the case where the number of the second key frames is 3, it is necessary to respectively determine whether the five map points have corresponding feature points on the three second key frame images.
  • step 402 the weight value corresponding to the first map point in the current second key frame image is determined to be 0, and the corresponding weight value of the second map point in the current second key frame image is determined.
  • the weight value is proportional to the target value.
  • the first map point does not have a corresponding feature point in the current second key frame image, and the corresponding weight value of the first map point in the current second key frame image is 0.
  • the second map point has a corresponding feature point on the current second key frame image, and the corresponding weight value of the second map point in the current second key frame image is proportional to a target value, and can increase with the increase of the target value .
  • the total number of third key frame images is determined.
  • the key frame image can be used as the third key frame image, and the embodiment of the present disclosure can determine the corresponding key frame image according to the total number of third key frame images.
  • the total number of third key frame images can be directly used as the target value.
  • the current second key frame image includes two second map points, map point 1 and map point 2, that is, map point 1 and map point 2 are maps with corresponding feature points on the current second key frame image point.
  • map point 1 has corresponding feature points on 3 key frame images
  • map point 2 has corresponding feature points on 4 key frame images.
  • the target value corresponding to map point 1 is 3, and the target value corresponding to map point 2 is 4.
  • the corresponding weight value of map point 1 on the current second key frame image is smaller than that of map point 2 on the current second key frame image. corresponding weight value.
  • the value of the weight value v ij of the map point on the constructed three-dimensional map can be determined by the following formula 1-4:
  • ⁇ can be 20, and D(X j ) is the target value corresponding to the map point X j .
  • the total number of third key frame images may be determined in the existing key frame images, so that the target value is determined according to the total number of third key frame images.
  • the weight value corresponding to the map point on the constructed three-dimensional map in the current second key frame image increases as the total number of third key frame images corresponding to the map point increases, so that the reconstructed map is more accurate.
  • the three-dimensional map reconstruction can be initialized based on the existing key frame images, so as to construct a three-dimensional map, and after a new key frame image is determined, new map points can be added to the constructed three-dimensional map.
  • the process of reconstruction initialization and building a 3D map may include the following steps:
  • step 501 the feature points included in each key frame image in the existing key frame images are determined.
  • the most reasonable image pair can be determined, and a 3D map can be constructed according to the initialized image pair. Add new map points to the constructed 3D map.
  • the Scale-Invariant Feature Transform (SIFT) method can be used to extract image features from each key frame image in the existing key frame images, and determine each key frame image. Multiple feature points are included, and the descriptors corresponding to the feature points are calculated.
  • SIFT Scale-Invariant Feature Transform
  • step 502 any two key frame images in the existing key frame images are regarded as a group of image pairs, and feature points of any group of image pairs are matched to determine a matching result.
  • any two key frame images in the existing key frame images form a set of image pairs
  • the Approximate Nearest Neighbor (ANN) algorithm can be used to perform feature point analysis on any set of image pairs. match to get the matching result.
  • the matching result includes but is not limited to the number of feature point matching.
  • step 503 according to the matching result, after sorting the multiple sets of image pairs in a pre-specified order, the three-dimensional map reconstruction initialization is performed based on different image pairs in sequence.
  • the ordering may be performed in a pre-specified order according to the number of feature point matching. For example, sort in descending order of the number of feature point matches.
  • the sorting may also be performed in combination with the distribution of the feature points, for example, according to the order of the matching number of feature points from high to low, and the distribution of feature points from scattered to concentrated. For the multiple sets of image pairs obtained after sorting, try to reconstruct and initialize sequentially from front to back.
  • a three-dimensional map is constructed based on the initialized image pairs that are successfully initialized among the plurality of sets of image pairs.
  • the baseline of the current group of image pairs can be determined, and when the baseline is within the preset baseline range, the current group of image pairs is used as the initialization image pair, and the initialization process is successful.
  • the preset baseline range can be obtained according to the empirical value, and it cannot be too large or too small.
  • the camera pose of the camera in the world coordinate system can be determined according to each image, so as to obtain the optical center coordinates of the camera.
  • a set of images The pair can determine two optical center coordinates O1 and O2.
  • the coordinate values of the feature points in the space coordinate system can also be respectively based on the image pair matching. For example, the coordinate value of the matching feature points on the image pair in FIG.
  • connection between P and O1 The line and the connecting line between P and O2 form an included angle 141, and the median value of the included angles corresponding to all matching feature points in the current image pair can be used as the baseline corresponding to the current image pair.
  • a three-dimensional map may be constructed based on the initialization image whose reconstruction is initialized successfully.
  • the three-dimensional map reconstruction and initialization can be performed based on the existing key frame images, which improves the efficiency of the initialization process, is simple to implement, and has high usability.
  • the map construction method provided by the embodiments of the present disclosure can be applied to 3D modeling scenarios in the fields of 3D modeling, augmented reality, image processing, photography, games, animation, film and television, e-commerce, education, real estate, home decoration, and the like.
  • a user uploads a video sequence to the cloud, and the cloud backend can realize fast and efficient 3D model reconstruction based on the map construction method provided by the embodiment of the present disclosure, and timely feedback the reconstruction result to the user;
  • Vision-based high-precision map construction can be realized, and the constructed map can be applied to augmented reality navigation;
  • vision-based dense map and model construction can be realized, and the constructed dense map and model can be applied It is used in augmented reality tours, digitization of cultural relics, scene roaming, building ranging and other scenarios.
  • An embodiment of the present application provides a map construction method, which uses an incremental reconstruction method to construct a map according to an input image or video sequence.
  • it is decided whether to register the latest image or video as a keyframe based on the number of map points observed in the latest image or video, as well as the distance and image disparity between the latest image or video and the nearest keyframe image image, and uniformly sample the image features of each newly registered key frame image to reduce the scale of map points, reduce the complexity of subsequent optimization, and improve reconstruction efficiency.
  • FIG. 14B is a schematic diagram of the implementation flow of a map construction method provided by an embodiment of the present disclosure.
  • the map construction method provided by the embodiment of the present application mainly includes the following processes: reconstruction initialization 141 , key frame selection 142, feature uniformization 143, local/global optimization adjustment 144.
  • the implementation process of each process is described below:
  • Reconstruction initialization The incremental reconstruction process can be started through the reconstruction initialization, and the quality of the reconstruction initialization greatly affects the robustness of the reconstruction.
  • the length of the baseline between image pairs, feature matching and distribution of feature points should be considered during reconstruction initialization.
  • the image pairs can be sorted according to the number of image matching points in the image pairs and the distribution of image feature points, and then each image pair is initialized. When the baseline between the image pairs is within a reasonable range, it can be determined that the image pair is initialized successfully. .
  • Key frame selection refers to the selection of key frame images in the incremental reconstruction process.
  • the key frame selection considers the following three aspects: a) the number of visible map points in the image; b) the image and the nearest key frame 3) The image disparity between the image and the nearest key frame.
  • the number of map points visible in the image reflects the similarity of the scene.
  • the adjacent frames of the image and the loopback frame can observe similar scenes with the image, so the loopback frame may not be considered, and the key frame image is limited to be determined only based on the adjacent frames of the image.
  • the current image can be registered as a key frame image.
  • the current image can be registered as a key frame image.
  • Image parallax is mainly used to deal with the problem that the camera stays in a certain position for a long time, resulting in too small parallax between the collected images, resulting in poor quality of the reconstructed 3D map.
  • Feature homogenization improves the problem of over-concentration of image features, reduces the complexity of global optimization, and improves computing efficiency.
  • Feature points without map observations can be uniformly sampled.
  • the sampling radius is reduced; otherwise, the sampling radius is increased until the number of sampled feature points is within a reasonable range (such as 900 ⁇ 2000).
  • the time series structure information of the input data is fully utilized, and the key frame images of the image sequence or the video sequence are extracted under the condition of ensuring the continuity of the corresponding scene of the image/video sequence, which can reduce the data Redundancy, simplifying the operation scale, and greatly improving the reconstruction efficiency;
  • the image feature homogenization operation it can reduce the over-concentration of features, reduce the complexity of optimizing the 3D map, and reduce the probability of image misregistration, Therefore, the accuracy and efficiency of map reconstruction can be improved;
  • the map points generated by the map construction method provided by the embodiments of the present disclosure require less storage space, which is convenient for map storage and transmission.
  • the reconstruction efficiency of the map construction method provided by the embodiment of the present disclosure can be improved by more than 5 times in a large scene.
  • the present disclosure also provides device embodiments.
  • FIG. 15 is a schematic structural diagram of a map construction device provided by an embodiment of the present disclosure, and the device includes:
  • the first determining part 610 is configured to determine whether the newly registered image belongs to the key frame image according to the association feature between the newly registered image and the existing key frame image;
  • the map update part 620 is configured to add new map points on the constructed three-dimensional map based on the newly determined key frame image each time.
  • the first determination part includes: a third determination sub-part, configured to determine the newly registered image when the newly registered image does not belong to the loop closure frame corresponding to the existing key frame image The total number of target feature points included in the image; wherein, the target feature points are feature points that have corresponding map points on the constructed three-dimensional map; the fourth determination subsection is configured to When the total number of points is less than the preset number, it is determined that the newly registered image belongs to the key frame image.
  • the first determining part includes: a fifth determining sub-part, configured to respectively determine the first coordinates and the second coordinate; wherein, the first key frame image is the key frame image whose acquisition time is closest to the newly registered image among the existing key frame images; the sixth determination subsection is configured to determine the first coordinate the first distance value between the second coordinate and the second coordinate; the seventh determination subsection is configured to determine that the newly registered image belongs to the key frame image under the condition that the first distance value meets the preset distance requirement .
  • the preset distance requirement includes satisfying the corresponding relationship with the target distance value; the apparatus further includes: a second determining part configured to determine the camera respectively in the existing key frame image the third and fourth coordinates in the world coordinate system when collecting two temporally adjacent key frame images; the third determining part is configured to determine the third coordinate between the third coordinate and the fourth coordinate Two distance values; a fourth determination part, configured to determine the target distance value according to the second distance value.
  • the first determining part includes: an eighth determining sub-part configured to determine an image parallax between the newly registered image and a first key frame image; wherein the first key frame image is Among the existing key frame images, the acquisition time is closest to the key frame image of the newly registered image; the ninth determination sub-section is configured to determine that the newly registered image belongs to a situation where the parallax of the image is greater than a preset parallax. the keyframe image.
  • the map update part includes: an acquisition sub-section, configured to sample the newly determined feature points included in the key frame image to obtain a plurality of sampling points; a generation sub-section, configured to On the constructed three-dimensional map, new map points are generated according to the plurality of sampling points.
  • the obtaining sub-section includes: a first determining unit configured to determine, among the newly determined feature points included in the key frame image, that are different from map points on the constructed three-dimensional map a matched second feature point; an acquiring unit configured to sample the second feature point on the newly determined key frame image to obtain the plurality of sampling points.
  • the acquiring unit includes: an area dividing subunit, configured to divide a plurality of sampling areas on the newly determined key frame image according to a preset sampling radius; an acquiring subunit, configured to The second feature points included in the sampling regions are uniformly sampled to obtain the plurality of sampling points.
  • the obtaining subunit is further configured to sample the same number of the second feature points in each sampling area, and obtain the plurality of sampling points whose total number satisfies a preset interval range.
  • the apparatus further includes: a map optimization part configured to optimize the constructed three-dimensional map according to the at least one second key frame image.
  • the apparatus further includes: a fifth determination part, configured to use the key frame image newly determined each time as the second key frame image; or a sixth determination part, configured to The secondly determined multiple key frame images are used as multiple second key frame images.
  • the map optimization part includes: a tenth determination sub-part, configured to determine a weight value corresponding to at least one map point on the constructed three-dimensional map in each second key frame image;
  • the eleventh determination subsection is configured to determine the loss function corresponding to the constructed three-dimensional map based on the corresponding weight value of the at least one map point in each of the second key frame images;
  • the twelfth determination subsection is configured to In order to determine the camera pose parameter that minimizes the loss function;
  • a thirteenth determination subsection is configured to update the at least one map point on the constructed three-dimensional map based on the camera pose parameter.
  • the tenth determination subsection includes: a second determination unit configured to determine, in at least one map point on the constructed three-dimensional map, that there is no corresponding current second key frame image The first map point of the feature point and the second map point with the corresponding feature point; the third determination unit is configured to determine that the weight value corresponding to the first map point in the current second key frame image is 0 , determining that the weight value corresponding to the second map point in the current second key frame image is proportional to the target value.
  • the apparatus further includes: a seventh determination part configured to determine the total number of third key frame images in the existing key frame images; wherein the second map point is in the There are corresponding feature points on the third key frame image; an eighth determining part is configured to determine the target value corresponding to the second map point based on the total number of the third key frame images.
  • the apparatus further includes: a ninth determination part, configured to determine the feature points included in each key frame image in the existing key frame images; a matching part, configured to determine the existing key frame images Any two key frame images in the frame image are used as a group of image pairs, and the feature points of any group of image pairs are matched to determine the matching result; the map reconstruction initialization part is configured to, according to the matching result, perform matching for multiple groups of image pairs. After the image pairs are sorted in a pre-specified order, the three-dimensional map is reconstructed and initialized sequentially based on different image pairs; the map construction part is configured to construct a three-dimensional map based on the initialized image pairs that have been successfully reconstructed and initialized in the multiple groups of image pairs.
  • a ninth determination part configured to determine the feature points included in each key frame image in the existing key frame images
  • a matching part configured to determine the existing key frame images Any two key frame images in the frame image are used as a group of image pairs, and the feature points of any group of image pairs are matched
  • a "part" may be a part of a circuit, a part of a processor, a part of a program or software, etc., of course, a unit, a module or a non-modularity.
  • Embodiments of the present disclosure further provide a computer-readable storage medium, where the computer storage medium may be a volatile storage medium or a non-volatile storage medium; the storage medium stores a computer program, and the computer program is configured to execute any one of the above Some or all of the steps in the map construction method of .
  • embodiments of the present disclosure provide a computer program product, the computer program product comprising a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, Part or all of the steps of any one of the above-mentioned map construction methods are implemented.
  • the computer program product can be specifically implemented by hardware, software or a combination thereof.
  • the computer program product is embodied as a computer storage medium, and in some embodiments, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK) and the like.
  • An embodiment of the present disclosure further provides a computer program, the computer program includes computer-readable codes, and when the computer-readable codes are read and executed by a computer, a part of any one of the map construction methods described above is implemented or all steps.
  • An embodiment of the present disclosure further provides a map construction device, including: a processor; a memory configured to store executable instructions of the processor; wherein the processor is configured to call the executable instructions stored in the memory to implement any of the above Some or all of the steps of a map construction method described in an item.
  • FIG. 16 is a schematic diagram of a hardware structure of a map construction device according to an embodiment of the present disclosure.
  • the map construction device 710 includes a processor 711 , and may further include an input device 712 , an output device 713 and a memory 714 .
  • the input device 712, the output device 713, the memory 714 and the processor 711 are connected to each other through a bus.
  • Memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), or portable Read-only memory (compact disc read-only memory, CD-ROM), which is used to store related instructions and data.
  • RAM random access memory
  • ROM read-only memory
  • EPROM erasable programmable read only memory
  • CD-ROM compact disc read-only memory
  • Input means are used for inputting data and/or signals, and output means are used for outputting data and/or signals.
  • the output device and the input device can be independent devices or an integral device.
  • the processor may include one or more processors, such as one or more central processing units (CPUs).
  • processors such as one or more central processing units (CPUs).
  • CPUs central processing units
  • the CPU may be a single-core CPU, or a Multi-core CPU.
  • Memory is used to store program codes and data for network devices.
  • the processor is configured to call the program code and data in the memory to execute the steps in the above method embodiments.
  • the processor is configured to call the program code and data in the memory to execute the steps in the above method embodiments.
  • the processor is configured to call the program code and data in the memory to execute the steps in the above method embodiments.
  • the description in the method embodiment please refer to the description in the method embodiment.
  • Figure 16 only shows a simplified design of a map building device.
  • the map construction device may also include other necessary elements, including but not limited to any number of input/output devices, processors, controllers, memories, etc., and all map construction devices that can implement the embodiments of the present disclosure All fall within the protection scope of the present disclosure.
  • the embodiments of the present disclosure provide a map construction method and apparatus, device, storage medium, computer program product, and computer program, wherein the method includes: determining the whether the newly registered image belongs to a key frame image; each time a new map point is added to the constructed three-dimensional map based on the newly determined key frame image.
  • the reconstruction of the three-dimensional map can be realized, and the redundancy of image data in the process of incremental reconstruction of the three-dimensional map can be reduced, the reconstruction scale can be simplified, and the reconstruction efficiency can be improved.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • Software Systems (AREA)
  • Geometry (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Artificial Intelligence (AREA)
  • Remote Sensing (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computer Graphics (AREA)
  • Image Analysis (AREA)

Abstract

本公开提供了一种地图构建方法及装置、设备、存储介质、计算机程序产品，其中，该方法包括：根据新注册图像与已有关键帧图像之间的关联特征，确定所述新注册图像是否属于关键帧图像；每次基于新确定的所述关键帧图像，在已构建的三维地图上增加新的地图点。

Description

地图构建方法及装置、设备、存储介质、计算机程序产品
相关申请的交叉引用
本公开基于申请号为202011599715.5、申请日为2020年12月29日、申请名称为“地图构建方法及装置、存储介质”的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此引入本申请作为参考。
技术领域
本公开涉及但不限于计算机视觉领域，尤其涉及地图构建方法及装置、设备、存储介质、计算机程序产品、计算机程序。
背景技术
目前，三维重建技术在文物保护、定位导航、场景漫游、增强现实、工业测量等领域有广泛的应用前景。三维重建方法可以分为两类：增量重建方法和全局重建方法。
发明内容
本公开实施例提供了一种地图构建方法及装置、设备、存储介质、计算机程序产品、计算机程序，可以如下实现：
本公开实施例提供一种地图构建方法，所述方法由电子设备执行，所述方法包括：根据新注册图像与已有关键帧图像之间的关联特征，确定所述新注册图像是否属于关键帧图像；每次基于新确定的所述关键帧图像，在已构建的三维地图上增加新的地图点。
在一些实施例中，所述根据新注册图像与已有关键帧图像之间的关联特征，确定所述新注册图像是否属于关键帧图像，包括：确定所述新注册图像是否属于所述已有关键帧图像对应的回环帧；在所述新注册图像属于所述已有关键帧图像对应的回环帧的情况下，确定所述新注册图像属于所述关键帧图像。
在一些实施例中，所述根据新注册图像与已有关键帧图像之间的关联特征，确定所述新注册图像是否属于关键帧图像，包括：在所述新注册图像不属于所述已有关键帧图像对应的回环帧的情况下，确定所述新注册图像所包括的目标特征点的总数目；其中，所述目标特征点是在所述已构建的三维地图上存在对应的地图点的特征点；在所述目标特征点的总数目小于预设数目的情况下，确定所述新注册图像属于所述关键帧图像。
在一些实施例中，所述根据新注册图像与已有关键帧图像之间的关联特征，确定所述新注册图像是否属于关键帧图像，包括：分别确定相机在采集所述新注册图像以及第一关键帧图像时，在世界坐标系中的第一坐标和第二坐标；其中，所述第一关键帧图像是所述已有关键帧图像中采集时间最接近所述新注册图像的关键帧图像；确定所述第一坐标和所述第二坐标之间的第一距离值；在所述第一距离值满足预设距离要求的情况下，确定所述新注册图像属于所述关键帧图像。
在一些实施例中，所述预设距离要求包括满足与目标距离值的对应关系；所述方法还包括：在所述已有关键帧图像中，分别确定所述相机在采集时间相邻的两张关键帧图像时，在世界坐标系中的第三坐标和第四坐标；确定所述第三坐标和所述第四坐标之间 的第二距离值；根据所述第二距离值确定所述目标距离值。
在一些实施例中，所述根据新注册图像与已有关键帧图像之间的关联特征，确定所述新注册图像是否属于关键帧图像，包括：确定所述新注册图像与第一关键帧图像之间的图像视差；其中，所述第一关键帧图像是所述已有关键帧图像中采集时间最接近所述新注册图像的关键帧图像；在所述图像视差大于预设视差的情况下，确定所述新注册图像属于所述关键帧图像。
在一些实施例中，所述每次基于新确定的所述关键帧图像，在已构建的三维地图上增加新的地图点，包括：对新确定的所述关键帧图像所包括的特征点进行采样，获得多个采样点；在已构建的三维地图上，根据所述多个采样点生成新的地图点。
在一些实施例中，所述对新确定的所述关键帧图像所包括的特征点进行采样，获得多个采样点，包括：在新确定的所述关键帧图像所包括的特征点中，确定与已构建的所述三维地图上的地图点不匹配的第二特征点；在新确定的所述关键帧图像上，对所述第二特征点进行采样，获得所述多个采样点。
在一些实施例中，所述在新确定的所述关键帧图像上，对所述第二特征点进行采样，获得所述多个采样点，包括：在新确定的所述关键帧图像上，根据预设采样半径，划分多个采样区域；对每个采样区域所包括的所述第二特征点进行均匀采样，获得所述多个采样点。
在一些实施例中，所述对每个采样区域所包括的所述第二特征点进行均匀采样，获得所述多个采样点，包括：在所述每个采样区域内采样相同数目的所述第二特征点，获得总数目满足预设区间范围的所述多个采样点。
在一些实施例中，所述方法还包括：根据至少一张第二关键帧图像，对已构建的三维地图进行优化。
在一些实施例中，所述方法还包括：将每次新确定的所述关键帧图像作为所述第二关键帧图像；或将分多次确定的多张所述关键帧图像作为多张第二关键帧图像。
在一些实施例中，所述根据至少一张第二关键帧图像，对已构建的三维地图进行优化，包括：确定所述已构建的三维地图上的至少一个地图点在每张第二关键帧图像中对应的权重值；基于所述至少一个地图点在所述每张第二关键帧图像中对应的权重值，确定已构建的三维地图对应的损失函数；确定使得所述损失函数最小的相机位姿参数；基于所述相机位姿参数，更新所述已构建的三维地图上的所述至少一个地图点。
在一些实施例中，所述确定所述已构建的三维地图上的至少一个地图点在每张第二关键帧图像中对应的权重值，包括：在所述已构建的三维地图上的至少一个地图点中，确定在当前第二关键帧图像中不存在对应的特征点的第一地图点和存在对应的特征点的第二地图点；确定所述第一地图点在所述当前第二关键帧图像中对应的权重值为0，确定所述第二地图点在所述当前第二关键帧图像中对应的所述权重值与目标值成正比。
在一些实施例中，所述方法还包括：在所述已有关键帧图像中，确定第三关键帧图像的总数目；其中，所述第二地图点在所述第三关键帧图像上存在对应的特征点；基于所述第三关键帧图像的总数目，确定与所述第二地图点对应的所述目标值。
在一些实施例中，所述方法还包括：确定所述已有关键帧图像中每张关键帧图像所 包括的特征点；将所述已有关键帧图像中的任意两张关键帧图像作为一组图像对，对任意一组图像对的特征点进行匹配，确定匹配结果；根据所述匹配结果，对多组图像对按照预先指定的顺序排序后，顺序基于不同图像对进行三维地图的重建初始化；基于所述多组图像对中重建初始化成功的初始化图像对，构建三维地图。
本公开实施例提供一种地图构建装置，所述装置包括：第一确定部分，配置为根据新注册图像与已有关键帧图像之间的关联特征，确定所述新注册图像是否属于关键帧图像；地图更新部分，配置为每次基于新确定的所述关键帧图像，在已构建的三维地图上增加新的地图点。
本公开实施例提供一种计算机可读存储介质，所述存储介质存储有计算机程序，所述计算机程序配置为执行本公开实施例中所描述的地图构建方法的部分或全部步骤。
本公开实施例提供一种地图构建设备，包括：处理器；配置为存储所述处理器可执行指令的存储器；其中，所述处理器被配置为调用所述存储器中存储的可执行指令，实现本公开实施例中所描述的地图构建方法的部分或全部步骤。
本公开实施例提供一种计算机程序产品，所述计算机程序产品包括存储了计算机程序的非瞬时性计算机可读存储介质，所述计算机程序被计算机读取并执行时，实现本公开实施例中所描述的地图构建方法的部分或全部步骤。
本公开实施例提供一种计算机程序，所述计算机程序包括计算机可读代码，在所述计算机可读代码被计算机读取并执行的情况下，实现本公开实施例中所描述的地图构建方法的部分或全部步骤。
本公开的实施例提供的技术方案可以包括以下有益效果：
本公开实施例中，可以根据新注册图像与已有关键帧图像之间的关联特征，在新注册图像中选取关键帧图像，每次基于新确定的关键帧图像，在已构建的三维地图上增加新的地图点。本公开实施例减少了三维地图增量式重建过程中图像数据的冗余，简化了重建规模，提高了重建效率。
本公开实施例中，在新注册图像属于已有关键帧图像对应的回环帧的情况下，该新注册图像属于关键帧图像。本公开实施例将已有关键帧图像对应的回环帧作为比较重要的关键帧图像，提高了地图重建的准确性，便于保护地图重建场景。
本公开实施例中，在新注册图像不属于已有关键帧图像对应的回环帧的情况下，可以在已构建的三维地图上，确定新注册图像所包括的目标特征点的总数目。其中，目标特征点是在所述已构建的三维地图上存在对应的地图点的特征点。在目标特征点的总数目小于预设数目的情况下，说明新注册图像拍摄的当前场景未被已有的关键帧图像观测到，可以作为新确定的关键帧图像，从而减少了三维地图增量式重建过程中图像数据的冗余，简化了重建规模，提高了重建效率。
本公开实施例中，可以分别确定相机在采集新注册图像以及第一关键帧图像时，在世界坐标系中的第一坐标和第二坐标，其中，第一关键帧图像是已有关键帧图像中与新注册图像的采集时间最接近的一张关键帧图像。在本公开的一些实施例中，可以确定第一坐标和第二坐标之间的第一距离值，在所述第一距离值满足预设距离要求的情况下，可以确定所述新注册图像属于所述关键帧图像。本公开实施例中，可以根据相机采集新 注册图像和第一关键帧图像时，在世界坐标系中的位置，来确定新注册图像是否属于关键帧图像。减少了三维地图增量式重建过程中图像数据的冗余，简化了重建规模，提高了重建效率。
本公开实施例中，可以根据已有关键帧图像，确定相机在采集相邻两张关键帧图像时，在世界坐标系中的第三坐标和第四坐标，从而确定第三坐标和第四坐标之间的第二距离值。根据第二距离值确定目标距离值。在第一距离值超过上述目标距离值的情况下，可以确定第一距离满足预设距离要求，新注册图像属于关键帧图像。本公开实施例可以通过调整第一距离值与目标距离值的关系，控制关键帧图像的插入频率，减少三维地图增量式重建过程中图像数据的冗余，简化了重建规模，提高了重建效率。
本公开实施例中，还可以根据新注册图像与已有关键帧图像中采集时间最接近的第一关键帧图像之间的图像视差，来确定新注册图像是否属于关键帧图像。充分利用输入的注册图像的时序结构，在保证场景连接性的情况下，确定关键帧图像，每次可以根据新确定的关键帧图像，在已构建的三维地图上增加新的地图点。减少了三维地图增量式重建过程中图像数据的冗余，简化了重建规模，提高了重建效率。
本公开实施例中，可以对新确定的关键帧图像所包括的特征点进行采样，获得多个采样点。从而在已构建的三维地图上，根据多个采样点生成新的地图点。在本公开的一些实施例中，在新确定的关键帧图像上，针对在已构建的三维地图上没有对应的地图点的第二特征点进行均匀采样，通过调整采样半径，可以控制采样点的总数目。从而可以减少图像特征过于集中的情况，降低了后续优化时的复杂度，提高了重建效率。
本公开实施例中，可以根据至少一张第二关键帧图像，对已构建的三维地图进行优化。其中，可以将每次新确定的所述关键帧图像作为第二关键帧图像，从而在每次新确定关键帧图像之后，对已构建的三维地图进行局部优化。或者还可以将分多次确定的多张关键帧图像作为多张第二关键帧图像，对已构建的三维地图进行全局优化。降低了优化复杂度，提高了重建的准确性。
本公开实施例中，可以确定已构建的三维地图上的至少一个地图点，在每张第二关键帧图像中是否存在对应的特征点。在第一地图点在当前第二关键帧图像中不存在对应的特征点的情况下，确定第一地图点在当前第二关键帧图像中对应的权重值为0，确定存在对应的特征点的第二地图点，在当前第二关键帧图像中对应的权重值与目标值成正比。本公开实施例中，可以在已有的所述关键帧图像中，确定第三关键帧图像的总数目，根据第三关键帧图像的总数目确定目标值。从而让地图点在当前第二关键帧图像中对应的权重值，随着该地图点对应的第三关键帧图像的总数目的增多而提高，使得重建的地图更加准确，实现对已构建的三维地图进行优化。
应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，并不能限制本公开。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本公开的实施例，并与说明书一起用于解释本公开的原理。
图1是本公开实施例提供的一种地图构建方法的实现流程示意图；
图2是本公开实施例提供的一种地图构建方法的实现流程示意图；
图3是本公开实施例提供的一种地图构建方法的实现流程示意图；
图4是本公开实施例提供的一种地图构建方法的实现流程示意图；
图5是本公开实施例提供的一种地图构建方法的实现流程示意图；
图6是本公开实施例提供的一种地图构建方法的实现流程示意图；
图7是本公开实施例提供的一种地图构建方法的实现流程示意图；
图8是本公开实施例提供的一种对第二特征点进行均匀采样的场景示意图；
图9是本公开实施例提供的一种地图构建方法的实现流程示意图；
图10是本公开实施例提供的一种地图构建方法的实现流程示意图；
图11是本公开实施例提供的一种地图构建方法的实现流程示意图；
图12是本公开实施例提供的一种地图构建方法的实现流程示意图；
图13是本公开实施例提供的一种地图构建方法的实现流程示意图；
图14A是本公开实施例提供的一种确定基线的场景示意图；
图14B为本公开实施例提供的一种地图构建方法的实现流程示意图；
图15是本公开实施例提供的一种地图构建装置的组成结构示意图；
图16是本公开实施例提供的一种地图构建设备的组成结构示意图。
具体实施方式
这里将详细地对示例性实施例进行说明，其示例表示在附图中。下面的描述涉及附图时，除非另有表示，不同附图中的相同数字表示相同或相似的要素。以下示例性实施例中所描述的实施方式并不代表与本公开相一致的所有实施方式。相反，它们仅是与如发明内容中所详述的、本公开实施例所提供的装置和方法的例子。
在本公开实施例运行的术语是仅仅出于描述特定实施例的目的，而非旨在限制本公开。在本公开实施例和发明内容中所运行的单数形式的“一种”、“所述”和“该”也旨在包括多数形式，除非上下文清楚地表示其他含义。还应当理解，本文中运行的术语“和/或”是指并包含一个或多个相关联的列出项目的任何或所有可能组合。
应当理解，尽管在本公开可能采用术语第一、第二、第三等来描述各种信息，但这些信息不应限于这些术语。这些术语仅用来将同一类型的信息彼此区分开。例如，在不脱离本公开范围的情况下，第一信息也可以被称为第二信息，类似地，第二信息也可以被称为第一信息。取决于语境，如在此所运行的词语“如果”可以被解释成为“在……时”或“当……时”或“响应于确定”。
为了更好地理解本申请实施例提供的地图构建方法，下面先对相关技术中的增量重建方法和全局重建方法进行说明。
相关技术中，增量重建方法在重建小规模场景时，能够达到较高的重建精度。但是，随着场景规模的扩大，重建效率急剧下降，同时因为增量重建过程逐帧注册的方式，会导致大场景重建产生漂移现象。全局重建可以通过场景图直接推算相机位姿，只需执行一次全局调整过程，适用于大规模场景重建的任务。但全局重建对匹配噪声敏感，并且没有经过充分优化，因此重建的鲁棒性和精度有所下降。
例如图1所示，图1是本公开实施例提供的一种地图构建方法的实现流程示意图， 该方法可以由电子设备执行，电子设备可以是终端设备或服务器或其它处理设备，其中，终端设备可以为用户设备(User Equipment，UE)、移动设备、用户终端、终端、蜂窝电话、无绳电话、个人数字处理(Personal Digital Assistant，PDA)、手持设备、计算设备、车载设备、可穿戴设备等，服务器可以是物理服务器、虚拟机、云服务器、服务器集群等。在一些实现方式中，该方法可以通过处理器调用存储器中存储的计算机可读指令的方式来实现。如图1所示，该方法可以包括以下步骤：
在步骤101中，根据新注册图像与已有关键帧图像之间的关联特征，确定所述新注册图像是否属于关键帧图像。
在本公开实施例中，新注册图像是相机在移动过程中每次新采集到的图像。关键帧图像是从新注册图像中按照时序由前到后顺序筛选出来的比较重要的图像，在确保场景连续性的基础上，可以基于关键帧图像进行三维地图重建。
在步骤102中，每次基于新确定的所述关键帧图像，在已构建的三维地图上增加新的地图点。
在本公开实施例中，已构建的三维地图是基于已有关键帧图像进行三维重建初始化后，构建得到的三维地图。每次新确定一张关键帧图像之后，根据新确定的关键帧图像所包括的特征点，采用三维点匹配的方法在已构建的三维地图上增加新的三维地图点，从而实现三维地图的重建。
上述实施例中，可以根据新注册图像与已有关键帧图像之间的关联特征，在新注册图像中选取关键帧图像，每次基于新确定的关键帧图像，在已构建的三维地图上增加新的地图点。本公开实施例减少了三维地图增量式重建过程中图像数据的冗余，简化了重建规模，提高了重建效率。
在一些实施例中，可以采用以下方式中的至少一种确定新注册图像是否属于关键帧图像。
第一种方式，根据新注册图像是否属于已有关键帧图像对应的回环帧来确定新注册图像是否属于关键帧图像。
例如图2所示，步骤101可以包括：
在步骤101-11中，确定所述新注册图像是否属于所述已有关键帧图像对应的回环帧。
在本公开实施例中，回环帧是指在拍摄了某个场景之后，间隔一段时间又再次对该场景进行拍摄所得到的图像。回环帧是为了确保场景重建的准确性。因此，回环帧在三维重建过程中是很重要的。
本公开实施例中，可以根据新注册图像与已有关键帧图像之间的图像特征的匹配度，以及新注册图像与已有关键帧图像之间间隔的注册图像的数目来判断新注册图像是否属于已有关键帧图像对应的回环帧。
在新注册图像与已有关键帧图像之间的图像特征的匹配度超过预设匹配度，且间隔的注册图像的数目超过预设图像数目的情况下，可以认为新注册图像属于已有关键帧图像对应的回环帧。
在步骤101-12中，在所述新注册图像属于所述已有关键帧图像对应的回环帧的情 况下，确定所述新注册图像属于所述关键帧图像。
在本公开实施例中，直接将回环帧作为关键帧图像，后续可以根据关键帧图像进行三维地图重建。
上述实施例中，在新注册图像属于已有关键帧图像对应的回环帧的情况下，该新注册图像属于关键帧图像。本公开实施例将回环帧作为比较重要的关键帧图像，提高了地图重建的准确性，便于保护地图重建场景。
第二种方式，新注册图像不属于已有关键帧图像对应的回环帧的情况下，可以根据目标特征点的总数目，确定新注册图像是否属于关键帧图像。
例如图3所示，步骤101可以包括：
在步骤101-21中，在所述新注册图像不属于所述已有关键帧图像对应的回环帧的情况下，确定所述新注册图像所包括的目标特征点的总数目。
在本公开实施例中，在按照上述步骤101-11中的方式确定新注册图像不属于回环帧的情况下，可以采用三维点匹配方法，确定所述新注册图像所包括的每个特征点在已构建的三维地图上是否存在对应的地图点，在存在对应的地图点的情况下，该特征点就是目标特征点。本公开实施例可以确定新注册图像所包括的目标特征点的总数目。
在步骤101-22中，在所述目标特征点的总数目小于预设数目的情况下，确定所述新注册图像属于所述关键帧图像。
在本公开实施例中，在目标特征点的总数目小于预设数目的情况下，可以认为新注册图像所对应的场景未被已有关键帧图像观测到，新注册图像包括新场景，可以确定新注册图像属于所述关键帧图像。其中，预设数目可以是大于零的正整数，例如可以是300。
在上述实施例中，在新注册图像不属于已有关键帧图像对应的回环帧的情况下，那么可以在已构建的三维地图上，确定新注册图像所包括的目标特征点的总数目。其中，目标特征点是在所述已构建的三维地图上存在对应的地图点的特征点。在目标特征点的总数目小于预设数目的情况下，说明新注册图像拍摄的当前场景未被已有的关键帧图像观测到，可以作为新确定的关键帧图像，从而减少了三维地图增量式重建过程中图像数据的冗余，简化了重建规模，提高了重建效率。
第三种方式，根据新注册图像与已有关键帧图像之间的距离值，即相机移动的距离值，来确定新注册图像是否属于关键帧图像。例如图4所示，步骤101可以包括：
在步骤101-31中，分别确定相机在采集所述新注册图像以及第一关键帧图像时，在世界坐标系中的第一坐标和第二坐标。
在本公开实施例中，考虑到观测到相似场景时，相机在世界坐标系中的位置应该没有移动很远。因此，可以分别确定相机在采集所述新注册图像和第一关键帧图像时，在世界坐标系中的第一坐标和第二坐标。其中，第一关键帧图像是已有关键帧图像中采集时间最接近所述新注册图像的关键帧图像，即第一关键帧图像是与新注册图像采集时间最近的一张关键帧图像。第一坐标和第二坐标可以通过是相机在世界坐标系中的位姿来表示。
在本公开实施例中，可以根据新注册图像中特征点与对应的三维的地图点之间的对应关系，采用随机样本一致性(Random Sample Consensus，RANSAC)和透视n点 (Perspective-n-Points，P3P)算法估计出相机在采集新注册图像时，在世界坐标系中的位姿，从而确定第一坐标。采用同样的方式，也可以确定出相机在采集第一关键帧图像时，在世界坐标系中的第二坐标。
在步骤101-32中，确定所述第一坐标和所述第二坐标之间的第一距离值。
在本公开实施例中，在确定了第一坐标和第二坐标之后，可以在世界坐标系中计算出两者之间的第一距离值。
在步骤101-33中，在所述第一距离值满足预设距离要求的情况下，确定所述新注册图像属于所述关键帧图像。
在本公开实施例中，预设距离要求可以包括但不限于满足与目标距离值的对应关系，对应关系可以包括但不限于是目标距离值与大于0的预设数值的乘积关系。
在一个示例中，可以在已有关键帧图像中，先确定相机在采集时间相邻的两张关键帧图像时，在世界坐标系中的第三坐标和第四坐标，将第三坐标和第四坐标之间的第二距离值的平均值作为目标距离值。
在第一距离值满足上述预设距离要求的情况下，可以确定新注册图像属于所述关键帧图像。在一些实施例中，目标距离值
Figure PCTCN2021092114-appb-000001
可以用以下公式1-1表示：
Figure PCTCN2021092114-appb-000002
其中，C k表示相机在采集第k个关键帧图像时在世界坐标系中的坐标，f(k)表示距离第k个关键帧图像采集时间最近的关键帧图像索引，K是已有的关键帧图像的总数目，k为大于1的整数。
在确定新注册图像是否属于关键帧图像时，可以采用以下公式1-2来确定：
Figure PCTCN2021092114-appb-000003
其中，C k+1表示相机在采集新注册图像时，在世界坐标系中的第一坐标，f(k+1)表示第一关键帧图像的索引，第一关键帧图像是已有关键帧图像中与新注册图像采集时间最接近的关键帧图像，C f(k+1)表示相机在采集第一关键帧图像时，在世界坐标系中的第二坐标，
Figure PCTCN2021092114-appb-000004
为目标距离值，σ是大于0的预设倍数。
通过公式1-2可以看出，通过调整σ可以控制关键帧图像插入频率，当σ≥1时，公式1-2就成为单调递增函数，关键帧图像之间的间隔会越来越大。当σ＜1时，可以让关键帧图像的插入频率保持在一个固定范围。在本公开实施例中，可以让σ位于[0.8，1]的区间范围内。
需要说明的是，第三种方式可以单独实施，也可以在上述步骤101-11，确定新注册图像不属于已有关键帧图像对应的回环帧的基础上，根据相机移动距离值来确定新注册图像是否属于关键帧图像。
上述本公开实施例中，可以通过调整预设倍数，控制关键帧图像的插入频率，有利于减少三维地图增量式重建过程中图像数据的冗余，简化了重建规模，提高了重建效率。
第四种方式，根据新注册图像与已有关键帧图像之间的图像视差来确定新注册图像是否属于关键帧图像。例如图5所示，步骤101可以包括：
在步骤101-41中，确定所述新注册图像与第一关键帧图像之间的图像视差。
在本公开实施例中，第一关键帧图像就是已有关键帧图像中采集时间最接近新注册 图像的关键帧图像。图像视差是指从两个不同的点上观察同一个目标所产生的方向差异，一般可以通过视差角来衡量，即从同一个目标看两个不同的点时对应确定的夹角。
在本公开实施例中，可以从已构建的三维地图上选取一个地图点，该地图点在新注册图像和第一关键帧图像上均存在对应的特征点。在世界坐标系中，确定该地图点与新注册图像上对应的特征点之间的连线，以及该地图点与第一关键帧图像上对应的特征点之间的连线的夹角，将这个夹角作为视差角，用来衡量新注册图像与第一关键帧图像之间的图像视差。
在步骤101-42中，在所述图像视差大于预设视差的情况下，确定所述新注册图像属于所述关键帧图像。
在本公开实施例中，预设视差可以根据经验证确定，例如可以为大于等于零的值。
需要说明的是，第四种方式可以单独实施，也可以在上述步骤101-11，确定新注册图像不属于已有关键帧图像对应的回环帧的基础上，根据图像视差来确定新注册图像是否属于关键帧图像。或者还可以在确定新注册图像不属于已有关键帧图像对应的回环帧的基础上，结合第三种方式，根据相机移动距离值来共同确定新注册图像是否属于关键帧图像，本公开实施例对此不作限定。例如，可以在确定新注册图像不属于已有关键帧图像对应的回环帧的基础上，确定相机移动距离值满足预设距离要求，且与第一关键帧图像之间的图像视差大于预设视差的情况下，确定该新注册图像属于关键帧图像。
上述实施例中，可以改善相机长时间停留在某个位置，导致采集的图像之间的视差过小，从而造成重建的三维地图质量变差的问题。
在一些实施例中，例如图6所示，步骤102可以包括：
在步骤102-1中，对新确定的所述关键帧图像所包括的特征点进行采样，获得多个采样点。
在本公开实施例中，可以针对每次新确定的关键帧图像所包括的特征点进行均匀采样，以减少图像特征过于集中的情况，得到多个采样点。
在步骤102-2中，在已构建的三维地图上，根据所述多个采样点生成新的地图点。
将多个采样点通过三维点匹配的方式，在已构建的三维地图上，生成新的地图点。
在一些实施例中，例如图7所示，步骤102-1可以包括：
在步骤201中，在新确定的所述关键帧图像所包括的特征点中，确定与已构建的所述三维地图上的地图点不匹配的第二特征点。
在本公开实施例中，新确定的关键帧图像上包括多个特征点，可以确定每个特征点在已构建的地图上是否存在匹配的地图点，在存在匹配的地图点的情况下，该特征点作为第一特征点，否则将该特征点作为第二特征点。
例如图8所示，已有的关键帧图像包括图像1和图像2，其中，图像1和图像2中的第一特征点82在已有的三维地图中存在匹配的地图点81。新确定的关键帧图像为图像3，在图像3的多个特征点中，在某个特征点在已有的三维地图中存在匹配的地图点的情况下，该特征点对应第一特征点82，否则对应第二特征点83。
在步骤202中，在新确定的所述关键帧图像上，对所述第二特征点进行采样，获得所述多个采样点。
上述实施例中，可以在新确定的关键帧图像所包括的特征点中，确定与已构建的三维地图上的地图点不匹配的第二特征点，后续基于第二特征点进行采样，以便生成新的地图点，减少生成与已有地图点重复的地图点，可用性高。
在一些实施例中，例如图9所示，步骤102-12可以包括：
在步骤301中，在新确定的所述关键帧图像上，根据预设采样半径，划分多个采样区域。
在本公开实施例中，预设采样半径可以根据采样点总数目的需要来划分。在总数目较多的情况下，可以减少预设采样半径，反之，增加预设采样半径，从而在新确定的关键帧图像上可以划分出多个采样区域，例如图8所示，在图像3中可以划分出采样区域1、采样区域2和采样区域3。
在步骤302中，对每个采样区域所包括的所述第二特征点进行均匀采样，获得所述多个采样点。
为了确保地图的连续性，可以在每个采样区域内对第二特征点进行均匀采样，采样相同数目的所述第二特征点，获得总数目满足预设区间范围的所述多个采样点。预设区域范围可以是[900，2000]之间的任意整数。
例如图8所示，假设在每个采样区域内采样一个第二特征点，那么在采样区域1和2中只保留一个第二特征点。在本公开实施例中，可以保留更接近采样区域中心位置的第二特征点。在采样区域3中由于只有一个第二特征点，那么保留该第二特征点。
上述实施例中，可以在新确定的关键帧图像上，针对在已构建的三维地图上没有对应的地图点的第二特征点进行均匀采样，通过调整采样半径，可以控制采样点的总数目。减少图像特征过于集中的情况，降低了后续优化时的复杂度，提高了重建效率。
在一些实施例中，例如图10所示，上述方法还可以包括：
在步骤103中，根据至少一张第二关键帧图像，对已构建的三维地图进行优化。
在本公开实施例中，可以将每次新确定的关键帧图像作为第二关键帧图像，每次新确定一张关键帧图像，就对三维地图进行局部优化。还可以将分多次确定的多张所述关键帧图像作为多张第二关键帧图像，对三维地图进行全局优化。
在一些实施例中，例如图11所示，步骤103可以包括：
在步骤103-1中，确定所述已构建的三维地图上的至少一个地图点在每张第二关键帧图像中对应的权重值。
在本公开实施例中，可以根据每个地图点在每张第二关键帧图像上是否存在对应的特征点，来确定该地图点在每张第二关键帧图像中对应的权重值。
在步骤103-2中，基于所述至少一个地图点在所述每张第二关键帧图像中对应的权重值，确定已构建的三维地图对应的损失函数。
在本公开实施例中，损失函数f(*)可以通过以下公式1-3表示：
Figure PCTCN2021092114-appb-000005
其中，n是已有的关键帧图像的总数目，m是已构建的三维地图所包括的地图点的总数目，K是相机内参，一般是预先标定好的，R i和T i是相机在采集第i张关键帧图像时对应的旋转矩阵和平移向量，一般可以通过相机位姿来确定。X j是已构建的三维地 图上的第j个地图点。x j是第j个地图点在关键帧图像上对应的特征点。v ij是第j个地图点在第i张关键帧图像上对应的特征点的权重值。
在步骤103-3中，确定使得所述损失函数最小的相机位姿参数。
上述进行局部或全局优化的目的就是为了让f(*)这个函数值减小。相机位姿参数包括但不限于上述公式1-3中的R i和T i。
在步骤103-4中，基于所述相机位姿参数，更新所述已构建的三维地图上的所述至少一个地图点。
在本公开实施例中，可以重新确定已构建的三维地图上的每个地图点所对应的相机位姿参数，从而基于重新确定的相机位姿参数，更新已构建的三维地图上的地图点，实现对已构建的三维地图进行全局优化或局部优化。
在一些实施例中，例如图12所示，步骤103-1可以包括：
在步骤401中，在所述已构建的三维地图上的至少一个地图点中，确定在当前第二关键帧图像中不存在对应的特征点的第一地图点和存在对应的特征点的第二地图点。在本公开实施例中，已构建的三维地图上的至少一个地图点需要分别在每张第二关键帧图像中确定是否存在对应的特征点。从而得到在当前第二关键帧图像上不存在对应的特征点的第一地图点，以及存在对应的特征点的第二地图点。
例如，已构建的三维地图上存在5个地图点，第二关键帧图像的数目为1，则需要分别确定5个地图点在该第二关键帧图像上是否存在对应的特征点。在第二关键帧的数目为3的情况下，需要分别确定5个地图点在三张第二关键帧图像上是否存在对应的特征点。
在步骤402中，确定所述第一地图点在所述当前第二关键帧图像中对应的权重值为0，确定所述第二地图点在所述当前第二关键帧图像中对应的所述权重值与目标值成正比。
在本公开实施例中，第一地图点在当前第二关键帧图像中不存在对应的特征点，第一地图点在当前第二关键帧图像中对应的权重值就是0。
第二地图点在当前第二关键帧图像上存在对应的特征点，第二地图点在当前第二关键帧图像中对应的权重值与一个目标值成正比，可以随着目标值的增加而增加。
在一个示例中，在所有的所述已有关键帧图像中，确定第三关键帧图像的总数目。在第二地图点在某张关键帧图像上存在对应的特征点的情况下，该关键帧图像可以作为第三关键帧图像，本公开实施例可以根据第三关键帧图像的总数目，确定与第二地图点对应的目标值。在一些实施例中，可以直接将第三关键帧图像的总数目，作为目标值。
例如，在当前第二关键帧图像上包括两个第二地图点，地图点1和地图点2，即地图点1和地图点2是在当前第二关键帧图像上存在对应的特征点的地图点。已有关键帧图像共有5张，地图点1在其中的3张关键帧图像上存在对应的特征点，地图点2在其中的4张关键帧图像上存在对应的特征点。则地图点1对应的目标值为3，地图点2对应的目标值为4，相应地，地图点1在当前第二关键帧图像上对应的权重值小于地图点2在当前第二关键帧图像上对应的权重值。
已构建的三维地图上地图点的权重值v ij的值可以采用以下公式1-4来确定：
Figure PCTCN2021092114-appb-000006
但是，由于地图点X j的权重值为0或者1，没有侧重，可能导致地图优化的局限性较高，影响优化的准确性。
在本公开的一些实施例中，还可以采用以下公式1-5来确定v ij的值：
Figure PCTCN2021092114-appb-000007
其中，ω可以为20，D(X j)位地图点X j对应的目标值。
上述实施例中，可以在已有关键帧图像中，确定第三关键帧图像的总数目，从而根据第三关键帧图像的总数目确定所述目标值。使得已构建的三维地图上的地图点在当前第二关键帧图像中对应的权重值，随着该地图点对应的第三关键帧图像的总数目的增多而提高，使得重建的地图更加准确。
在一些实施例中，可以基于已有关键帧图像进行三维地图重建初始化，从而构建三维地图，后续在确定了新的关键帧图像后，可以在已构建的三维地图上添加新的地图点。例如图13所示，重建初始化以及构建三维地图的过程可以包括以下步骤：
在步骤501中，确定所述已有关键帧图像中每张关键帧图像所包括的特征点。
在本公开实施例中，在进行三维重建之前需要先对多张已注册图像进行重建初始化，可以确定基线最合理的图像对，根据初始化图像对构建三维地图，每次出现新注册图像之后，在构建的三维地图上增加新的地图点。
在多张已注册图像中，可以采用尺度不变特征变换(Scale-Invariant Feature Transform，SIFT)的方法，对已有关键帧图像中的每张关键帧图像提取图像特征，确定每张关键帧图像所包括的多个特征点，并计算特征点对应的描述子。
在步骤502中，将所述已有关键帧图像中的任意两张关键帧图像作为一组图像对，对任意一组图像对的特征点进行匹配，确定匹配结果。
在本公开实施例中，已有关键帧图像中的任意两张关键帧图像组成一组图像对，可以采用近似最近邻(Approximate Nearest Neighbor，ANN)算法对任意一组图像对进行特征点匹配，得到匹配结果。其中，匹配结果包括但不限于特征点匹配数目。
在步骤503中，根据所述匹配结果，对多组图像对按照预先指定的顺序排序后，顺序基于不同图像对进行三维地图的重建初始化。
在本公开实施例中，针对多组图像对，可以根据特征点匹配数目按照预先指定的顺序进行排序。例如，按照特征点匹配数目由高到低的顺序进行排序。在一些实施例中，还可以结合特征点分布情况进行排序，例如按照特征点匹配数目由高到低，且特征点分布由分散到集中的顺序进行排序。对排序后得到的多组图像对，由前到后依次尝试进行重建初始化。
在步骤504中，基于所述多组图像对中重建初始化成功的初始化图像对，构建三维地图。
在初始化过程中，可以确定当前一组图像对的基线，在该基线位于预设基线范围内的情况下，将当前这组图像对作为初始化图像对，初始化过程成功。预设基线范围可以 根据经验值得到，不能太大也不能太小。
其中，在确定当前一组图像对的基线时，可以根据每张图像分别确定相机在世界坐标系中的相机位姿，从而得到相机的光心坐标，例如图14A所示，一组图像对可以确定两个光心坐标O1和O2。另外，还可以分别根据图像对匹配的特征点在空间坐标系中的坐标值，例如图14A中图像对上的匹配特征点在世界坐标系的坐标值为P，那么P与O1之间的连线、和P与O2之间的连线构成一个夹角141，当前图像对中所有匹配的特征点对应的夹角的中值可以作为当前图像对对应的基线。
在本公开实施例中，可以基于重建初始化成功的初始化图像，构建三维地图。
上述实施例中，可以基于已有关键帧图像进行三维地图重建初始化，提高了初始化过程的效率，实现简便，可用性高。
本公开实施例提供的地图构建方法可以应用于三维建模、增强现实、图像处理、摄影、游戏、动画、影视、电子商务、教育、房地产、家居装修等领域中的三维建模场景。例如，用户上传视频序列到云端，云端后台可以基于本公开实施例提供的地图构建方法实现快速高效的三维模型重建，并及时将重建结果反馈给用户；通过本公开实施例提供的地图构建方法，可以实现基于视觉的高精度地图构建，构建的地图可以应用于增强现实导航；通过本公开实施例提供的地图构建方法，可以实现基于视觉的稠密地图和模型构建，构建的稠密地图和模型可以应用于增强现实导览、文物数字化、场景漫游、建筑物测距等场景。
本申请实施例提供一种地图构建方法，根据输入的图像或视频序列，采用增量重建的方式进行地图构建。在增量重建过程中，根据最新图像或视频中观测到的地图点数量，以及最新图像或视频与最近的关键帧图像之间的距离和图像视差，决定是否将最新图像或视频注册为关键帧图像，并对每次新注册的关键帧图像的图像特征进行均匀采样，以减少地图点规模，降低后续优化时的复杂度，提高重建效率。
参见图14B，图14B为本公开实施例提供的一种地图构建方法的实现流程示意图，如图14B所示，本申请实施例提供的地图构建方法主要包括如下流程：重建初始化141、关键帧选取142、特征均匀化143、局部/全局优化调整144。下面分别说明每一流程的实现过程：
1)重建初始化：通过重建初始化可以启动增量重建流程，重建初始化的质量很大程度影响了重建的鲁棒性。重建初始化时需考虑图像对之间的基线的长度、特征匹配和特征点的分布情况。可以根据图像对中的图像匹配点数和图像特征点的分布对图像对进行排序，然后对每一图像对进行初始化，当图像对之间的基线位于合理范围内时，可以确定该图像对初始化成功。
2)关键帧选取：关键帧选取是指选取增量重构过程中的关键帧图像，关键帧选取考虑如下三个方面：a)图像中可视的地图点数量；b)图像与最近关键帧的距离；3)图像与最近关键帧的图像视差。图像中可视的地图点数量反映了场景的相似程度，当新注册图像中的特征点大部分都能在全局地图中找到对应的三维点时，说明当前场景已被其他注册过的关键帧图像观测到。实际上，图像的相邻帧和回环帧均能与该图像观测到相似场景，可以不考虑回环帧，限定关键帧图像只基于图像的相邻帧进行确定。此外， 还可以设定当前图像中可视的地图点数量少于300个点时，将当前图像注册为关键帧图像。同样地，两个图像观测到相似场景时，相应的观测点之间的空间距离不会太远。在当前图像满足上述公式1-2的情况下，可以将当前图像注册为关键帧图像。图像视差主要用来处理相机长时间停留在某个位置，导致采集的图像之间的视差过小，从而造成重建的三维地图质量变差的问题。
3)特征均匀化；特征均匀化改善了图像特征过于集中的问题，可以降低全局优化时的复杂度，从而可以提升运算效率。为了保证地图的连续性，可以只对没有地图观测的特征点均匀采样。采样时，每个采样区域只保留一个特征点，采样的特征点数量小于特定阈值的情况下，减小采样半径；反之，增大采样半径，直到采样的特征点数量在合理区间(如900～2000)。
4)局部/全局优化调整：可以采用前文中公式1-3所示的损失函数，对相机的位姿参数进行优化，从而实现三维地图的局部/全局优化调整。
本公开实施例提供的地图构建方法，首先，充分利用输入数据的时序结构信息，在保证图像/视频序列对应场景的连续性的情况下，抽取图像序列或视频序列的关键帧图像，可以减少数据冗余，简化运算规模，极大地提升了重建效率；其次，通过图像特征均匀化操作，可以减少特征过于集中的情况，降低对三维地图进行优化的复杂度，且能减少图像错误注册的概率，从而可以提升地图重建精度和效率；最后，在保证地图场景结构特征不变的情况下，本公开实施例提供的地图构建方法生成的地图点需要更少的存储空间，便于地图存储和传输。与相关技术中的地图构建方法相比，本公开实施例提供的地图构建方法在大场景下的重建效率可以提升5倍以上。
与前述方法实施例相对应，本公开还提供了装置的实施例。
如图15所示，图15是本公开实施例提供的一种地图构建装置的组成结构示意图，装置包括：
第一确定部分610，配置为根据新注册图像与已有关键帧图像之间的关联特征，确定所述新注册图像是否属于关键帧图像；
地图更新部分620，配置为每次基于新确定的所述关键帧图像，在已构建的三维地图上增加新的地图点。
在一些实施例中，所述第一确定部分包括：第三确定子部分，配置为在所述新注册图像不属于所述已有关键帧图像对应的回环帧的情况下，确定所述新注册图像所包括的目标特征点的总数目；其中，所述目标特征点是在所述已构建的三维地图上存在对应的地图点的特征点；第四确定子部分，配置为在所述目标特征点的总数目小于预设数目的情况下，确定所述新注册图像属于所述关键帧图像。
在一些实施例中，所述第一确定部分包括：第五确定子部分，配置为分别确定相机在采集所述新注册图像以及第一关键帧图像时，在世界坐标系中的第一坐标和第二坐标；其中，所述第一关键帧图像是所述已有关键帧图像中采集时间最接近所述新注册图像的关键帧图像；第六确定子部分，配置为确定所述第一坐标和所述第二坐标之间的第一距离值；第七确定子部分，配置为在所述第一距离值满足预设距离要求的情况下，确定所述新注册图像属于所述关键帧图像。
在一些实施例中，所述预设距离要求包括满足与目标距离值的对应关系；所述装置还包括：第二确定部分，配置为在所述已有关键帧图像中，分别确定所述相机在采集时间相邻的两张关键帧图像时，在世界坐标系中的第三坐标和第四坐标；第三确定部分，配置为确定所述第三坐标和所述第四坐标之间的第二距离值；第四确定部分，配置为根据所述第二距离值确定所述目标距离值。
在一些实施例中，所述第一确定部分包括：第八确定子部分，配置为确定所述新注册图像与第一关键帧图像之间的图像视差；其中，所述第一关键帧图像是所述已有关键帧图像中采集时间最接近所述新注册图像的关键帧图像；第九确定子部分，配置为在所述图像视差大于预设视差的情况下，确定所述新注册图像属于所述关键帧图像。
在一些实施例中，所述地图更新部分包括：获取子部分，配置为对新确定的所述关键帧图像所包括的特征点进行采样，获得多个采样点；生成子部分，配置为在已构建的三维地图上，根据所述多个采样点生成新的地图点。
在一些实施例中，所述获取子部分包括：第一确定单元，配置为在新确定的所述关键帧图像所包括的特征点中，确定与已构建的所述三维地图上的地图点不匹配的第二特征点；获取单元，配置为在新确定的所述关键帧图像上，对所述第二特征点进行采样，获得所述多个采样点。
在一些实施例中，所述获取单元包括：区域划分子单元，配置为在新确定的所述关键帧图像上，根据预设采样半径，划分多个采样区域；获取子单元，配置为对每个采样区域所包括的所述第二特征点进行均匀采样，获得所述多个采样点。
在一些实施例中，所述获取子单元还配置为在所述每个采样区域内采样相同数目的所述第二特征点，获得总数目满足预设区间范围的所述多个采样点。
在一些实施例中，所述装置还包括：地图优化部分，配置为根据至少一张第二关键帧图像，对已构建的三维地图进行优化。
在一些实施例中，所述装置还包括：第五确定部分，配置为将每次新确定的所述关键帧图像作为所述第二关键帧图像；或第六确定部分，配置为将分多次确定的多张所述关键帧图像作为多张第二关键帧图像。
在一些实施例中，所述地图优化部分包括：第十确定子部分，配置为确定所述已构建的三维地图上的至少一个地图点在每张第二关键帧图像中对应的权重值；第十一确定子部分，配置为基于所述至少一个地图点在所述每张第二关键帧图像中对应的权重值，确定已构建的三维地图对应的损失函数；第十二确定子部分，配置为确定使得所述损失函数最小的相机位姿参数；第十三确定子部分，配置为基于所述相机位姿参数，更新所述已构建的三维地图上的所述至少一个地图点。
在一些实施例中，所述第十确定子部分包括：第二确定单元，配置为在所述已构建的三维地图上的至少一个地图点中，确定在当前第二关键帧图像中不存在对应的特征点的第一地图点和存在对应的特征点的第二地图点；第三确定单元，配置为确定所述第一地图点在所述当前第二关键帧图像中对应的权重值为0，确定所述第二地图点在所述当前第二关键帧图像中对应的所述权重值与目标值成正比。
在一些实施例中，所述装置还包括：第七确定部分，配置为在所述已有关键帧图像 中，确定第三关键帧图像的总数目；其中，所述第二地图点在所述第三关键帧图像上存在对应的特征点；第八确定部分，配置为基于所述第三关键帧图像的总数目，确定与所述第二地图点对应的所述目标值。
在一些实施例中，所述装置还包括：第九确定部分，配置为确定所述已有关键帧图像中每张关键帧图像所包括的特征点；匹配部分，配置为将所述已有关键帧图像中的任意两张关键帧图像作为一组图像对，对任意一组图像对的特征点进行匹配，确定匹配结果；地图重建初始化部分，配置为根据所述匹配结果，对多组图像对按照预先指定的顺序排序后，顺序基于不同图像对进行三维地图的重建初始化；地图构建部分，配置为基于所述多组图像对中重建初始化成功的初始化图像对，构建三维地图。
对于装置实施例而言，由于其基本对应于方法实施例，所以相关之处参见方法实施例的部分说明即可。以上所描述的装置实施例仅仅是示意性的，其中作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部部分来实现本公开实施例方案的目的。本领域普通技术人员在不付出创造性劳动的情况下，即可以理解并实施。
在本公开实施例以及其他的实施例中，“部分”可以是部分电路、部分处理器、部分程序或软件等等，当然也可以是单元，还可以是模块也可以是非模块化的。
本公开实施例还提供了一种计算机可读存储介质，计算机存储介质可为易失性存储介质或非易失性存储介质；存储介质存储有计算机程序，计算机程序配置为执行上述任一所述的地图构建方法的部分或全部步骤。
在一些实施例中，本公开实施例提供了一种计算机程序产品，所述计算机程序产品包括存储了计算机程序的非瞬时性计算机可读存储介质，所述计算机程序被计算机读取并执行时，实现上述任一所述的地图构建方法的部分或全部步骤。
该计算机程序产品可以具体通过硬件、软件或其结合的方式实现。在一些实施例中，所述计算机程序产品具体体现为计算机存储介质，在一些实施例中，计算机程序产品具体体现为软件产品，例如软件开发包(Software Development Kit，SDK)等等。
本公开实施例还提供一种计算机程序，所述计算机程序包括计算机可读代码，在所述计算机可读代码被计算机读取并执行的情况下，实现上述任一所述的地图构建方法的部分或全部步骤。
本公开实施例还提供了一种地图构建设备，包括：处理器；配置为存储处理器可执行指令的存储器；其中，处理器被配置为调用所述存储器中存储的可执行指令，实现上述任一项所述的地图构建方法的部分或全部步骤。
图16为本公开实施例提供的一种地图构建设备的硬件结构示意图。该地图构建设备710包括处理器711，还可以包括输入装置712、输出装置713和存储器714。该输入装置712、输出装置713、存储器714和处理器711之间通过总线相互连接。
存储器包括但不限于是随机存储记忆体(random access memory，RAM)、只读存储器(read-only memory，ROM)、可擦除可编程只读存储器(erasable programmable read only memory，EPROM)、或便携式只读存储器(compact disc read-only memory，CD-ROM)， 该存储器用于存储相关指令及数据。
输入装置用于输入数据和/或信号，以及输出装置用于输出数据和/或信号。输出装置和输入装置可以是独立的器件，也可以是一个整体的器件。
处理器可以包括是一个或多个处理器，例如包括一个或多个中央处理器(central processing unit，CPU)，在处理器是一个CPU的情况下，该CPU可以是单核CPU，也可以是多核CPU。
存储器用于存储网络设备的程序代码和数据。
处理器配置为调用该存储器中的程序代码和数据，执行上述方法实施例中的步骤。具体可参见方法实施例中的描述。
可以理解的是，图16仅仅示出了一种地图构建设备的简化设计。在实际应用中，地图构建设备还可以分别包含必要的其他元件，包含但不限于任意数量的输入/输出装置、处理器、控制器、存储器等，而所有可以实现本公开实施例的地图构建设备都在本公开的保护范围之内。
本领域技术人员在考虑说明书及实践这里公开的发明后，将容易想到本公开的其它实施方案。本公开旨在涵盖本公开的任何变型、用途或者适应性变化，这些变型、用途或者适应性变化遵循本公开的一般性原理并包括本公开未公开的本技术领域中的公知常识或者惯用技术手段。说明书和实施例仅被视为示例性描述。
以上所述仅为本公开的部分实施例而已，并不用以限制本公开，凡在本公开的精神和原则之内，所做的任何修改、等同替换、改进等，均应包含在本公开保护的范围之内。
工业实用性
本公开实施例提供了一种地图构建方法及装置、设备、存储介质、计算机程序产品、计算机程序，其中，该方法包括：根据新注册图像与已有关键帧图像之间的关联特征，确定所述新注册图像是否属于关键帧图像；每次基于新确定的所述关键帧图像，在已构建的三维地图上增加新的地图点。根据本公开实施例，可以实现三维地图的重建，并能减少三维地图增量式重建过程中图像数据的冗余，简化重建规模，提高重建效率。

Claims (36)

  1. 一种地图构建方法，所述方法由电子设备执行，包括：
    根据新注册图像与已有关键帧图像之间的关联特征，确定所述新注册图像是否属于关键帧图像；
    每次基于新确定的所述关键帧图像，在已构建的三维地图上增加新的地图点。
  2. 根据权利要求1所述的方法，其中，所述根据新注册图像与已有关键帧图像之间的关联特征，确定所述新注册图像是否属于关键帧图像，包括：
    确定所述新注册图像是否属于所述已有关键帧图像对应的回环帧；
    在所述新注册图像属于所述已有关键帧图像对应的回环帧的情况下，确定所述新注册图像属于所述关键帧图像。
  3. 根据权利要求1或2所述的方法，其中，所述根据新注册图像与已有关键帧图像之间的关联特征，确定所述新注册图像是否属于关键帧图像，包括：在所述新注册图像不属于所述已有关键帧图像对应的回环帧的情况下，确定所述新注册图像所包括的目标特征点的总数目；其中，所述目标特征点是在所述已构建的三维地图上存在对应的地图点的特征点；在所述目标特征点的总数目小于预设数目的情况下，确定所述新注册图像属于所述关键帧图像。
  4. 根据权利要求1-3任一项所述的方法，其中，所述根据新注册图像与已有关键帧图像之间的关联特征，确定所述新注册图像是否属于关键帧图像，包括：
    分别确定相机在采集所述新注册图像以及第一关键帧图像时，在世界坐标系中的第一坐标和第二坐标；其中，所述第一关键帧图像是所述已有关键帧图像中采集时间最接近所述新注册图像的关键帧图像；确定所述第一坐标和所述第二坐标之间的第一距离值；在所述第一距离值满足预设距离要求的情况下，确定所述新注册图像属于所述关键帧图像。
  5. 根据权利要求4所述的方法，其中，所述预设距离要求包括满足与目标距离值的对应关系；所述方法还包括：在所述已有关键帧图像中，分别确定所述相机在采集时间相邻的两张关键帧图像时，在世界坐标系中的第三坐标和第四坐标；确定所述第三坐标和所述第四坐标之间的第二距离值；根据所述第二距离值确定所述目标距离值。
  6. 根据权利要求1-5任一项所述的方法，其中，所述根据新注册图像与已有关键帧图像之间的关联特征，确定所述新注册图像是否属于关键帧图像，包括：确定所述新注册图像与第一关键帧图像之间的图像视差；其中，所述第一关键帧图像是所述已有关键帧图像中采集时间最接近所述新注册图像的关键帧图像；在所述图像视差大于预设视差的情况下，确定所述新注册图像属于所述关键帧图像。
  7. 根据权利要求1-6任一项所述的方法，其中，所述每次基于新确定的所述关键帧图像，在已构建的三维地图上增加新的地图点，包括：
    对新确定的所述关键帧图像所包括的特征点进行采样，获得多个采样点；
    在已构建的三维地图上，根据所述多个采样点生成新的地图点。
  8. 根据权利要求7所述的方法，其中，所述对新确定的所述关键帧图像所包括的特征点进行采样，获得多个采样点，包括：在新确定的所述关键帧图像所包括的特征点 中，确定与已构建的所述三维地图上的地图点不匹配的第二特征点；在新确定的所述关键帧图像上，对所述第二特征点进行采样，获得所述多个采样点。
  9. 根据权利要求8所述的方法，其中，所述在新确定的所述关键帧图像上，对所述第二特征点进行采样，获得所述多个采样点，包括：
    在新确定的所述关键帧图像上，根据预设采样半径，划分多个采样区域；
    对每个采样区域所包括的所述第二特征点进行均匀采样，获得所述多个采样点。
  10. 根据权利要求9所述的方法，其中，所述对每个采样区域所包括的所述第二特征点进行均匀采样，获得所述多个采样点，包括：在所述每个采样区域内采样相同数目的所述第二特征点，获得总数目满足预设区间范围的所述多个采样点。
  11. 根据权利要求1-10任一项所述的方法，其中，所述方法还包括：
    根据至少一张第二关键帧图像，对已构建的三维地图进行优化。
  12. 根据权利要求11所述的方法，其中，所述方法还包括：
    将每次新确定的所述关键帧图像作为所述第二关键帧图像；或
    将分多次确定的多张所述关键帧图像作为多张第二关键帧图像。
  13. 根据权利要求11或12所述的方法，其中，所述根据至少一张第二关键帧图像，对已构建的三维地图进行优化，包括：确定所述已构建的三维地图上的至少一个地图点在每张第二关键帧图像中对应的权重值；基于所述至少一个地图点在所述每张第二关键帧图像中对应的权重值，确定已构建的三维地图对应的损失函数；确定使得所述损失函数最小的相机位姿参数；基于所述相机位姿参数，更新所述已构建的三维地图上的所述至少一个地图点。
  14. 根据权利要求13所述的方法，其中，所述确定所述已构建的三维地图上的至少一个地图点在每张第二关键帧图像中对应的权重值，包括：
    在所述已构建的三维地图上的至少一个地图点中，确定在当前第二关键帧图像中不存在对应的特征点的第一地图点和存在对应的特征点的第二地图点；
    确定所述第一地图点在所述当前第二关键帧图像中对应的权重值为0，确定所述第二地图点在所述当前第二关键帧图像中对应的所述权重值与目标值成正比。
  15. 根据权利要求14所述的方法，其中，所述方法还包括：在所述已有关键帧图像中，确定第三关键帧图像的总数目；其中，所述第二地图点在所述第三关键帧图像上存在对应的特征点；基于所述第三关键帧图像的总数目，确定与所述第二地图点对应的所述目标值。
  16. 根据权利要求1-15任一项所述的方法，其中，所述方法还包括：
    确定所述已有关键帧图像中每张关键帧图像所包括的特征点；将所述已有关键帧图像中的任意两张关键帧图像作为一组图像对，对任意一组图像对的特征点进行匹配，确定匹配结果；根据所述匹配结果，对多组图像对按照预先指定的顺序排序后，顺序基于不同图像对进行三维地图的重建初始化；基于所述多组图像对中重建初始化成功的初始化图像对，构建三维地图。
  17. 一种地图构建装置，所述装置包括：
    第一确定部分，配置为根据新注册图像与已有关键帧图像之间的关联特征，确定所述新注册图像是否属于关键帧图像；地图更新部分，配置为每次基于新确定的所述关 键帧图像，在已构建的三维地图上增加新的地图点。
  18. 根据权利要求17所述的装置，其中，所述第一确定部分包括：第一确定子部分，配置为确定所述新注册图像是否属于所述已有关键帧图像对应的回环帧；第二确定子部分，配置为在所述新注册图像属于所述已有关键帧图像对应的回环帧的情况下，确定所述新注册图像属于所述关键帧图像。
  19. 根据权利要求17或18所述的装置，其中，所述第一确定部分包括：第三确定子部分，配置为在所述新注册图像不属于所述已有关键帧图像对应的回环帧的情况下，确定所述新注册图像所包括的目标特征点的总数目；其中，所述目标特征点是在所述已构建的三维地图上存在对应的地图点的特征点；第四确定子部分，配置为在所述目标特征点的总数目小于预设数目的情况下，确定所述新注册图像属于所述关键帧图像。
  20. 根据权利要求17-19中任一项所述的装置，其中，所述第一确定部分包括：第五确定子部分，配置为分别确定相机在采集所述新注册图像以及第一关键帧图像时，在世界坐标系中的第一坐标和第二坐标；其中，所述第一关键帧图像是所述已有关键帧图像中采集时间最接近所述新注册图像的关键帧图像；第六确定子部分，配置为确定所述第一坐标和所述第二坐标之间的第一距离值；第七确定子部分，配置为在所述第一距离值满足预设距离要求的情况下，确定所述新注册图像属于所述关键帧图像。
  21. 根据权利要求20所述的装置，其中，所述预设距离要求包括满足与目标距离值的对应关系；所述装置还包括：第二确定部分，配置为在所述已有关键帧图像中，分别确定所述相机在采集时间相邻的两张关键帧图像时，在世界坐标系中的第三坐标和第四坐标；第三确定部分，配置为确定所述第三坐标和所述第四坐标之间的第二距离值；第四确定部分，配置为根据所述第二距离值确定所述目标距离值。
  22. 根据权利要求17-21中任一项所述的装置，其中，所述第一确定部分包括：第八确定子部分，配置为确定所述新注册图像与第一关键帧图像之间的图像视差；其中，所述第一关键帧图像是所述已有关键帧图像中采集时间最接近所述新注册图像的关键帧图像；第九确定子部分，配置为在所述图像视差大于预设视差的情况下，确定所述新注册图像属于所述关键帧图像。
  23. 根据权利要求17-21中任一项所述的装置，其中，所述地图更新部分包括：获取子部分，配置为对新确定的所述关键帧图像所包括的特征点进行采样，获得多个采样点；生成子部分，配置为在已构建的三维地图上，根据所述多个采样点生成新的地图点。
  24. 根据权利要求23所述的装置，其中，所述获取子部分包括：第一确定单元，配置为在新确定的所述关键帧图像所包括的特征点中，确定与已构建的所述三维地图上的地图点不匹配的第二特征点；获取单元，配置为在新确定的所述关键帧图像上，对所述第二特征点进行采样，获得所述多个采样点。
  25. 根据权利要求24所述的装置，其中，所述获取单元包括：区域划分子单元，配置为在新确定的所述关键帧图像上，根据预设采样半径，划分多个采样区域；获取子单元，配置为对每个采样区域所包括的所述第二特征点进行均匀采样，获得所述多个采样点。
  26. 根据权利要求25所述的装置，其中，所述获取子单元还配置为在所述每个采 样区域内采样相同数目的所述第二特征点，获得总数目满足预设区间范围的所述多个采样点。
  27. 根据权利要求17-26中任一项所述的装置，其中，所述装置还包括：地图优化部分，配置为根据至少一张第二关键帧图像，对已构建的三维地图进行优化。
  28. 根据权利要求27所述的装置，其中，所述装置还包括：第五确定部分，配置为将每次新确定的所述关键帧图像作为所述第二关键帧图像；或第六确定部分，配置为将分多次确定的多张所述关键帧图像作为多张第二关键帧图像。
  29. 根据权利要求27或28所述的装置，其中，所述地图优化部分包括：第十确定子部分，配置为确定所述已构建的三维地图上的至少一个地图点在每张第二关键帧图像中对应的权重值；第十一确定子部分，配置为基于所述至少一个地图点在所述每张第二关键帧图像中对应的权重值，确定已构建的三维地图对应的损失函数；第十二确定子部分，配置为确定使得所述损失函数最小的相机位姿参数；第十三确定子部分，配置为基于所述相机位姿参数，更新所述已构建的三维地图上的所述至少一个地图点。
  30. 根据权利要求29所述的装置，其中，所述第十确定子部分包括：第二确定单元，配置为在所述已构建的三维地图上的至少一个地图点中，确定在当前第二关键帧图像中不存在对应的特征点的第一地图点和存在对应的特征点的第二地图点；第三确定单元，配置为确定所述第一地图点在所述当前第二关键帧图像中对应的权重值为0，确定所述第二地图点在所述当前第二关键帧图像中对应的所述权重值与目标值成正比。
  31. 根据权利要求30所述的装置，其中，所述装置还包括：第七确定部分，配置为在所述已有关键帧图像中，确定第三关键帧图像的总数目；其中，所述第二地图点在所述第三关键帧图像上存在对应的特征点；第八确定部分，配置为基于所述第三关键帧图像的总数目，确定与所述第二地图点对应的所述目标值。
  32. 根据权利要求17-31中任一项所述的装置，其中，所述装置还包括：第九确定部分，配置为确定所述已有关键帧图像中每张关键帧图像所包括的特征点；匹配部分，配置为将所述已有关键帧图像中的任意两张关键帧图像作为一组图像对，对任意一组图像对的特征点进行匹配，确定匹配结果；地图重建初始化部分，配置为根据所述匹配结果，对多组图像对按照预先指定的顺序排序后，顺序基于不同图像对进行三维地图的重建初始化；地图构建部分，配置为基于所述多组图像对中重建初始化成功的初始化图像对，构建三维地图。
  33. 一种计算机可读存储介质，所述存储介质存储有计算机程序，所述计算机程序用于执行上述权利要求1-16任一项所述的地图构建方法。
  34. 一种地图构建设备，包括：处理器；用于存储所述处理器可执行指令的存储器；其中，所述处理器被配置为调用所述存储器中存储的可执行指令，实现权利要求1-16中任一项所述的地图构建方法。
  35. 一种计算机程序产品，所述计算机程序产品包括存储了计算机程序的非瞬时性计算机可读存储介质，所述计算机程序被计算机读取并执行时，实现如权利要求1-16中任一项所述的地图构建方法。
  36. 一种计算机程序，包括计算机可读代码，在所述计算机可读代码被计算机读取并执行的情况下，实现如权利要求1-16中任一项所述的地图构建方法。
PCT/CN2021/092114 2020-12-29 2021-05-07 地图构建方法及装置、设备、存储介质、计算机程序产品 WO2022142049A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202011599715.5A CN112614185B (zh) 2020-12-29 2020-12-29 地图构建方法及装置、存储介质
CN202011599715.5 2020-12-29

Publications (1)

Publication Number Publication Date
WO2022142049A1 true WO2022142049A1 (zh) 2022-07-07

Family

ID=75249118

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/092114 WO2022142049A1 (zh) 2020-12-29 2021-05-07 地图构建方法及装置、设备、存储介质、计算机程序产品

Country Status (2)

Country Link
CN (1) CN112614185B (zh)
WO (1) WO2022142049A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115655291A (zh) * 2022-09-28 2023-01-31 重庆中科汽车软件创新中心 激光slam闭环建图的方法、装置、移动机器人、设备及介质
CN116958267A (zh) * 2023-09-21 2023-10-27 腾讯科技（深圳）有限公司 位姿处理方法、装置、电子设备及存储介质
CN117953117A (zh) * 2022-10-20 2024-04-30 滴图(北京)科技有限公司 用于场景重建的方法、装置、设备、存储介质和程序产品

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112614185B (zh) * 2020-12-29 2022-06-21 浙江商汤科技开发有限公司 地图构建方法及装置、存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103247075A (zh) * 2013-05-13 2013-08-14 北京工业大学 基于变分机制的室内环境三维重建方法
US20180040137A1 (en) * 2015-09-17 2018-02-08 Skycatch, Inc. Generating georeference information for aerial images
CN109816769A (zh) * 2017-11-21 2019-05-28 深圳市优必选科技有限公司 基于深度相机的场景地图生成方法、装置及设备
CN110610520A (zh) * 2019-08-29 2019-12-24 中德（珠海）人工智能研究院有限公司 一种基于双球幕相机的视觉定位方法及系统
CN111968129A (zh) * 2020-07-15 2020-11-20 上海交通大学 具有语义感知的即时定位与地图构建系统及方法
CN112614185A (zh) * 2020-12-29 2021-04-06 浙江商汤科技开发有限公司 地图构建方法及装置、存储介质

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103413322B (zh) * 2013-07-16 2015-11-18 南京师范大学 序列视频关键帧提取方法
CN106846467B (zh) * 2017-01-23 2020-05-05 阿依瓦（北京）技术有限公司 基于每个相机位置优化的实体场景建模方法和系统
CN107369183A (zh) * 2017-07-17 2017-11-21 广东工业大学 面向mar的基于图优化slam的跟踪注册方法及系统
CN110276826A (zh) * 2019-05-23 2019-09-24 全球能源互联网研究院有限公司 一种电网作业环境地图的构建方法及系统
CN111325842B (zh) * 2020-03-04 2023-07-28 Oppo广东移动通信有限公司 地图构建方法、重定位方法及装置、存储介质和电子设备
CN111795704B (zh) * 2020-06-30 2022-06-03 杭州海康机器人技术有限公司 一种视觉点云地图的构建方法、装置

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103247075A (zh) * 2013-05-13 2013-08-14 北京工业大学 基于变分机制的室内环境三维重建方法
US20180040137A1 (en) * 2015-09-17 2018-02-08 Skycatch, Inc. Generating georeference information for aerial images
CN109816769A (zh) * 2017-11-21 2019-05-28 深圳市优必选科技有限公司 基于深度相机的场景地图生成方法、装置及设备
CN110610520A (zh) * 2019-08-29 2019-12-24 中德（珠海）人工智能研究院有限公司 一种基于双球幕相机的视觉定位方法及系统
CN111968129A (zh) * 2020-07-15 2020-11-20 上海交通大学 具有语义感知的即时定位与地图构建系统及方法
CN112614185A (zh) * 2020-12-29 2021-04-06 浙江商汤科技开发有限公司 地图构建方法及装置、存储介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
CUI, LIYANG: "Autonomous Robot Navigation System Based on Embedded Vision", MASTER THESIS, 1 May 2017 (2017-05-01), CN, pages 1 - 76, XP009538134 *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115655291A (zh) * 2022-09-28 2023-01-31 重庆中科汽车软件创新中心 激光slam闭环建图的方法、装置、移动机器人、设备及介质
CN117953117A (zh) * 2022-10-20 2024-04-30 滴图(北京)科技有限公司 用于场景重建的方法、装置、设备、存储介质和程序产品
CN116958267A (zh) * 2023-09-21 2023-10-27 腾讯科技（深圳）有限公司 位姿处理方法、装置、电子设备及存储介质
CN116958267B (zh) * 2023-09-21 2024-01-12 腾讯科技（深圳）有限公司 位姿处理方法、装置、电子设备及存储介质

Also Published As

Publication number Publication date
CN112614185B (zh) 2022-06-21
CN112614185A (zh) 2021-04-06

Similar Documents

Publication Publication Date Title
WO2022142049A1 (zh) 地图构建方法及装置、设备、存储介质、计算机程序产品
WO2020259248A1 (zh) 基于深度信息的位姿确定方法、装置、介质与电子设备
JP2023540917A (ja) ３次元再構成及び関連インタラクション、測定方法及び関連装置、機器
WO2019205852A1 (zh) 确定图像捕捉设备的位姿的方法、装置及其存储介质
TWI745818B (zh) 視覺定位方法、電子設備及電腦可讀儲存介質
US20240046557A1 (en) Method, device, and non-transitory computer-readable storage medium for reconstructing a three-dimensional model
WO2020259481A1 (zh) 定位方法及装置、电子设备、可读存储介质
CN112927363B (zh) 体素地图构建方法及装置、计算机可读介质和电子设备
CN109658445A (zh) 网络训练方法、增量建图方法、定位方法、装置及设备
CN111127524A (zh) 一种轨迹跟踪与三维重建方法、系统及装置
Irschara et al. Towards wiki-based dense city modeling
WO2021136386A1 (zh) 数据处理方法、终端和服务器
CN111784776B (zh) 视觉定位方法及装置、计算机可读介质和电子设备
JP7280393B2 (ja) 視覚位置決め方法、関連モデルの訓練方法及び関連装置並びに機器
CN111833447A (zh) 三维地图构建方法、三维地图构建装置及终端设备
WO2022033111A1 (zh) 图像信息提取方法、训练方法及装置、介质和电子设备
CN111323024A (zh) 定位方法及装置、设备、存储介质
CN107610219B (zh) 三维场景重构中几何线索感知的像素级点云稠密化方法
CN114565728A (zh) 地图构建方法、位姿确定方法及相关装置、设备
Jung et al. Efficient mobile AR technology using scalable recognition and tracking based on server-client model
WO2023221790A1 (zh) 图像编码器的训练方法、装置、设备及介质
CN109961103B (zh) 特征提取模型的训练方法、图像特征的提取方法及装置
CN114882106A (zh) 位姿确定方法和装置、设备、介质
US11127199B2 (en) Scene model construction system and scene model constructing method
CN112270748A (zh) 基于图像的三维重建方法及装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21912810

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21912810

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 15.02.2024)

122 Ep: pct application non-entry in european phase

Ref document number: 21912810

Country of ref document: EP

Kind code of ref document: A1