KR20230127830A - Methods performed by electronic devices, electronic devices and storage media - Google Patents

Abstract

Embodiments of the present disclosure provide a method performed by an electronic device, an electronic device, and a computer readable storage medium, and relate to a technical field performed by an electronic device. The method includes obtaining a search image for a query image; acquiring spatial features of each of the query image and the search image; estimating a relative pose between the query image and the search image based on the spatial image. A method executed by an electronic device provided by an embodiment of the present disclosure may determine a relative pose in an artificial intelligence manner and more accurately optimize a global map.

Description

Methods performed by electronic devices, electronic devices and storage media}

The present disclosure relates to the field of simultaneous localization and mapping (SLAM) technology, and relates to a method performed by an electronic device, the electronic device, and a computer readable storage medium.

SLAM is a technology that uses sensors such as cameras and laser radars on the device to create and describe a 3D map of the space where the device is located in real time, and to determine the pose (position and posture) of the device. Due to the error of camera calibration and the limitation of feature matching accuracy, unavoidable cumulative errors occur in the process of mapping and estimating the position of visual SLAM. To solve this problem, a closed-loop (LC) module is implemented in the SLAM system to implement drift-free localization by identifying common view relationships between the current frame and the initial key frame and optimizing the global map to reduce cumulative errors. can be added

Currently, related technologies generally use a method of optimizing a global map by constructing visual constraints through a method such as feature matching and then calculating a relative pose between a query image and a search image, but this method has relatively large visual changes. The problem that the time consumed for global map optimization is also relatively long cannot be solved, so it is necessary to optimize the current SLAM closed-loop module.

It is to provide a method performed by an electronic device, an electronic device and a computer readable storage medium. The technical problem to be achieved by this embodiment is not limited to the above technical problems, and other technical problems can be inferred from the following embodiments.

The present disclosure provides a method performed by an electronic device, an electronic device, and a computer readable storage medium, and a technical solution thereof is as follows.

In an embodiment, a method performed by an electronic device of the present disclosure includes obtaining a search image for a query image; acquiring spatial features of each of the query image and the search image; and estimating a relative pose between the query image and the search image based on the spatial feature.

In another embodiment, an electronic device of the present disclosure includes one or more processors; Memory; and one or more application programs, wherein the one or more application programs are stored in a memory and are configured to be executed by one or more processes, and the one or more application programs correspond to the method executed by the electronic device according to the embodiment. is configured to run

In another embodiment, a computer readable storage medium of the present disclosure stores at least one instruction, at least one program, code set, or instruction set, and at least one instruction, at least one program, code set, or instruction set. is loaded and executed by the process to implement the method executed by the electronic device as described in the above embodiment.

The technical effects provided by the present disclosure are as follows.

The present disclosure provides a method executed by an electronic device, an electronic device, and a computer-readable storage medium, and compared to the prior art, the present disclosure provides a relative pose between a query image and a search image by a spatial feature of the query image and the search image. By estimating , the spatial feature has a wider sensed field of view level and spatial information, and the global map can be more precisely optimized.

In addition, since the present disclosure estimates a set of three-dimensional points by densely and uniformly extracting keypoints and ORB descriptors from an image, and then completing stereo matching and triangulation using epipolar constraints, estimation 3D point sets are more uniformly and densely distributed in space than the global map. This allows a more accurate determination of the relative pose to optimize the global map.

In addition, the present disclosure can effectively optimize the global map through gradual cluster adjustment and overall cluster adjustment, reducing operation time and increasing accuracy.

In order to more clearly explain the technical solutions described in the embodiments of the present disclosure, drawings used in describing the embodiments of the present disclosure will be briefly introduced.
1 is a flowchart of a method performed by an electronic device provided by an embodiment of the present disclosure.
2 is a schematic diagram illustrating a method for clustering a 3D point set according to an embodiment of the present disclosure.
3 is a schematic diagram illustrating a method of generating a first feature matching pair and a second feature matching pair according to an embodiment of the present disclosure.
4 is a schematic diagram illustrating a method for generating a third feature matching pair according to an embodiment of the present disclosure.
5 is a schematic diagram illustrating a method for generating an optimized global map according to an embodiment of the present disclosure.
6 is a schematic structural diagram of an electronic device provided in an embodiment of the present disclosure.
7 is a schematic diagram illustrating a method performed by an electronic device provided in an embodiment of the present disclosure.
8 is a schematic structural diagram of an electronic device provided in an embodiment of the present disclosure.
9 is a schematic structural diagram of an electronic device provided in an embodiment of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described in detail. Examples of the above embodiments are shown in the drawings, wherein the same or similar reference numerals denote the same or similar components or components having the same or similar functions. Hereinafter, the embodiments described with reference to the drawings are only for explaining the present disclosure as illustrative, and do not limit the present disclosure.

Unless otherwise specified, singular forms such as “a” and “one” as used herein may also include plural forms. Also, use of the word "comprises" in the specification of this disclosure indicates the presence of such features, integers, steps, operations, parts and/or components, and one or more other features, integers, steps, operations, parts, It will be understood that it is not intended to exclude the presence or additional possibilities of elements and/or combinations thereof.

When a component is said to be "connected" or "coupled" to another component, it is understood that it may be directly connected or coupled to the other component, or other intermediate components may be present. Also, “connection” or “coupling” used herein may include wireless connection or wireless coupling. Words such as "and/or" as used herein include one or more interrelated enumerated items or any unit and in all combinations.

In order to more clearly clarify the purpose, technical solutions or advantages of the present disclosure, embodiments of the drawings will be described in more detail with reference to the following drawings.

SLAM is a technology that uses sensors such as cameras and inertial measurement units on the device to create a 3D map of the space where the device is located in real time, and to determine the location and posture of the device in real time on the map. The camera and inertial measurement unit are cheaper than LiDAR sensors, and are standard components of devices such as mobile phones, augmented reality glasses, and indoor robots, and can be used in various situations. In addition, the main research content of all existing SLAM technologies is to create maps and acquire device poses in real time using cameras and inertial measurement units as sensors. Compared to a monocular camera, a three-dimensional map built by a binocular camera has a more realistic physical scale, so in practical applications, the visual sensor on the device is often a binocular camera.

Existing SLAM systems are mainly based on multiple-view geometry theory, which uses tracking and matching of point features in an image to pose the device (the pose of the device represents the spatial 3-dimensional position and orientation of the device). ) and 3D environment information is acquired. In the time series, point features of video-related images are estimated and matched according to multi-view geometric principles, point features of binocular images are matched according to epipolar constraints, and finally, these matching are based on geometric constraints between device poses and 3D map points. The relationship is established, and the device's pose and 3D map points can be resolved through filtering or cluster adjustment.

Due to errors in camera calibration and feature matching, visual SLAM creates unavoidable cumulative errors in the process of map creation and localization. Implementing drift-free localization and creating an accurate global map are challenges to be solved. To solve this problem, a closed-loop (LC) module can be added to the SLAM system to implement drift-free localization by identifying the common view relationship between the current frame and the initial key frame and optimizing the global map to reduce the cumulative error. there is. Therefore, the LC constitutes a major module of the SLAM system, and can significantly improve SLAM performance.

LC is generally classified into three stages. The first step is similar to an image retrieval task, which aims to retrieve images that are semantically similar to the query image. Of course, proper image display is essential, and most methods are based on the Bag of Words (BoW) model. The second step is to form visual constraints through feature matching methods such as BoW and ORB (Oriented Fast and Rotated Binary Robust Independent Elementary Features) feature matching and projection matching, and then estimate the relative pose between the query image and the search image. will be. The third step is to optimize the global map to implement drift-free localization.

Recently, research on LC has been conducted. In some related technologies, four degrees of freedom (4DOF) pose graph optimization has been proposed in order to optimize global correspondence between a key frame pose and a current frame pose in a global map. The 4DOF pose graph optimization method optimizes the global consistency of key frame poses in a small amount of time. However, since one global map cannot be maintained, the accuracy of the optimization is reduced. In addition, in some related technologies, a technique of improving the LC recovery rate by replacing the temporal consistency check of three key frames with a local consistency check between a query key frame and three common view key frames has been proposed. However, if the camera's field of view change is relatively large and there is perceptual aliasing in the scene, the relative pose inlier between query and search key frames is assumed to be relatively small, and LC may also fail. Also, optimizing the global map with the whole bundle adjustment method (FBA) takes a relatively long time. In some other related technologies, a feature re-identification method is proposed, and the proposed global sub-map having temporal and spatial sensitivity in a prior pose helps quickly identify existing features. However, if the prior pose has low reliability, a drift-free camera pose is obtained by combining LC and feature re-identification. Also, if the camera drift is relatively large, feature re-identification does not work. Therefore, LC is also very likely to fail because of the large viewpoint change of the camera and perceptual aliasing in the scene. Also, if the drift of the camera is relatively large, the progressive bundle adjustment method (IBA) is not sufficient for the optimization of the global map.

Taken together, accurate and stable LCs present several challenges. First, feature matching uses local features in small blocks instead of hierarchical and spatial information with larger detection fields, such as ORB, BRIEF (Binary Robust Independent Elementary Features), SURF (Speeded Up Robust Features) and SIFT (Scale-invarient feature transform) , scale invariant feature transformation), etc., which can lead to LC instability in the presence of large field-of-view changes of the camera and perceptual aliasing in the scene. Feature matching based on deep learning typically focuses on learning better sparse detectors and local descriptors from data using convolutional neural networks (CNNs). Some recent work along this direction is matching neural networks of two sets of local features by jointly searching for correspondences and rejecting unmatchable points, which can solve a variety of multi-view geometrical problems requiring high-quality feature matching. . However, deep learning learning methods require powerful computing resources. Next, no single optimization method can reliably solve the global map optimization problem. For example, when the drift of the camera is large, IBA optimization is insufficient, optimizing the global map by the full bundle adjustment (FBA) method is very exhausting, and the pose optimization method cannot maintain an accurate global map.

Hereinafter, a technical method for solving the technical problem of the present disclosure will be described in detail through embodiments. The following embodiments may be combined with each other, and the same or similar concept or process may not be additionally described in some embodiments.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

Possible implementation methods are provided in the embodiments of the present disclosure, as shown in FIG. 1 , a method executed by an electronic device is provided. The method may include the following steps.

In step S101, a search image is acquired for the query image.

Here, the query image is an image collected by the electronic device during location estimation and map creation (eg, a scene image of a current frame). This may be an image received from another device.

In one embodiment, the query image may be collected in real time, the query image may be collected at regular intervals, or the query image may be collected by an event trigger. The process of acquiring the query image is not limited.

In one embodiment, in the process of estimating the position of an electronic device and creating a map, an image data set is configured for each key frame, an image data set corresponding to a query image is obtained, and a plurality of candidate images are included in the image data set. and search for data semantically similar to the query image from a plurality of candidate images to obtain a search image.

At this time, the number of search images may be one or several, but the present disclosure does not limit the number of search images.

For example, a search image may be retrieved from candidate images based on the BoW model.

In step S102, spatial features of the query image and spatial features of the search image are respectively acquired.

In this case, the spatial feature may include a 3D point set.

In some possible embodiments, the step of acquiring spatial features of each of the query image and the search image in step S102 is a step of acquiring spatial features for either one of the query image and the search image, including (1) an image keypoint and Extracting image feature points including feature descriptors; and (2) estimating a 3D point set by performing stereo matching on the image feature points.

At this time, the feature descriptor may be, for example, an ORB descriptor.

In an implementation process, a 3D point set of a query image and a 3D point set of a search image may be estimated respectively by performing stereo matching and triangulation using epipolar constraints.

In step S103, a relative pose between the query image and the search image is estimated based on the spatial feature.

In some possible embodiments, a feature matching result may be obtained by matching a spatial feature of a query image with a spatial feature between search images at least once, and a relative pose may be determined based on the feature matching result.

In an implementation process, coarse-to-fine matching of several layers is performed on the 3D point set of the query image and the 3D point set of the search image, and a relative pose may be determined based on the final matching result. The method for determining the relative pose is described in more detail below.

In any of the above embodiments, relative poses between the query image and the search image are estimated based on spatial features of the query image and the search image. Spatial features include larger layers of sense viewing angles and spatial information, allowing the global map to be optimized more accurately.

In some possible embodiments, a set of three-dimensional points is estimated by densely and uniformly extracting key points and ORB descriptors from the image, then completing stereo matching and triangulation using epipolar constraints. Because the three-dimensional point set is more uniformly and densely distributed in space than the global map, the global map can be optimized by more accurately determining relative poses.

Hereinafter, a process of determining a relative pose will be described in detail with reference to embodiments.

In some possible embodiments, the feature matching result includes a first feature matching pair, and the step of matching the spatial feature of the query image with the spatial feature of the search image at least once to obtain the feature matching result comprises a combination of the query image and the search image. and generating a first feature matching pair between a clustering result of a query image and a clustering result of a search image by clustering each 3D point set.

In some possible embodiments, the points of the three-dimensional point set are clustered into cubes according to their spatial distribution.

Central descriptor of each cluster is all 3-dimensional point descriptors in the cube the voting function of It is obtained by, and considers the spatial information of a larger sensing field of view.

in the expression, is the centroid descriptor of each cluster ego, is the 3D point descriptor of the cube, is the voting function.

In some possible embodiments, clustering each of the three-dimensional point sets of the query image and the search image to generate a first feature matching pair between the clustering result of the query image and the clustering result of the search image comprises the three-dimensional point set of the query image. Determining at least one first cube generated by clustering, determining at least one second cube formed by clustering a three-dimensional point set of a search image, determining a first cluster center of each of the first cubes, Determining a second cluster center of each second cube, and determining a second cluster center that matches the first cluster center of each first cube, and based on the first cluster center and the second cluster center that match each other and determining the first feature matching pair.

As in the embodiment shown in Figure 2, the number 1 is greater for the first dimension of the ORB descriptor described above. Thus, the first dimension of the cluster centroid descriptor is 1. After clustering a set of three-dimensional points, the cluster centroid descriptor of each cube can be obtained, ( ) is the cluster centroid descriptor of the query image, is the cluster centroid descriptor of the search image. Next, nearest neighbor search and mutual validation A coarse matching pair for a cube between a query image and a search image, that is, a first feature matching pair is obtained through

in the expression, ( ) is the cluster centroid descriptor of the query image, is the cluster centroid descriptor of the search image, represents the Hamming distance, represents the threshold of the Hamming distance, is the search for the nearest neighbor from the cluster centroid feature of the query image to the cluster centroid of the search image, is the search for nearest neighbors from the cluster centroid feature of the search image to the cluster centroid feature of the query image, is a verification of the cluster center feature of the query image and the cluster center feature of the search image.

In FIG. 3 , a cube connected by a dotted line of a double-headed arrow is a coarse matching pair between a query image and a search image.

In some possible embodiments, a first feature matching pair between cubes may be obtained through coarse matching of the query image and the search image, and a relative pose between the query image and the search image may be estimated directly based on the first feature matching pair. there is.

In some possible embodiments, after obtaining a first feature matching pair between cubes through coarse matching of a query image and a search image, fine matching may be performed again to obtain a second feature matching pair between 3D point sets. there is.

In an implementation process, the feature matching result may include a second feature matching pair, and obtaining a feature matching result by matching the spatial feature of the query image with the spatial feature of the search image at least once may include first feature matching. The method may further include obtaining a second feature matching pair between the 3D point set of the query image and the 3D point set of the search image by performing a nearest neighbor search and mutual verification on the 3D points of the pair region. Determining the relative pose based on a feature matching result may include estimating a relative position based on a second feature matching pair.

In other words, all three-dimensional points in the first feature matching pair region and Perform nearest neighbor search and mutual verification for class are respectively The set and limit of the 27 cubes of the spatial domain of the cubes It represents the set of 27 cubes among the spatial domains of the cubes. Next, a coarse relative pose between the query image and the search image based on the second feature matching pair is estimated

in the expression, silver ego, silver is, Is the nearest neighbor search from the 3D point feature of the query image to the 3D point feature of the search image, is the nearest neighbor search from the 3D point feature of the search image to the 3D point feature of the query image, is the verification of the 3D point features of the query image and the 3D point features of the search image.

In some possible embodiments, coarse matching is performed between the query image and the search image to obtain a first feature matching pair between cubes, fine matching is performed again to obtain a second feature matching pair between 3D point sets, A coarse relative pose between the query image and the search image may be estimated directly based on the second feature matching pair, and the coarse relative pose may be set as a relative pose between the query image and the search image.

In some possible embodiments, coarse matching is performed on the query image and the search image to obtain a first feature matching pair between cubes, fine matching is performed again to obtain a second feature matching pair between 3D point sets, and then again A third feature matching pair may be obtained by performing pose guide matching.

The feature matching result further includes a third feature matching pair, and the step of matching the spatial feature of the query image with the spatial feature of the search image at least once to obtain the feature matching result comprises querying based on the second feature matching pair. Estimating an approximate relative pose between the image and the search image, and projecting the 3D point set of the search image onto the coordinate system of the query image by the approximate relative pose, and the 3D point set of the query image and the 3D point set of the search image. The method may further include determining a third feature matching pair between point sets.

Determining the relative pose based on the feature matching result may include determining the relative position based on the third feature matching pair.

In the embodiment shown in Fig. 4, an approximate relative pose , project the 3D points of the search image onto the coordinate system of the query image, and then, similar to the fine matching part, perform nearest neighbor search and mutual verification according to the distance of the point location and the Hamming distance of the ORB descriptor to query the image. A third matching pair between the 3D point set of and the 3D point set of the search image is obtained, and finally, a prior relative pose between the query image and the search image is estimated based on the third feature pair. As shown in FIG. 4 , an overlapping portion exists between corresponding 3D points of a query image and a search image to form a third matching pair, and a 3D point that does not overlap at all represents an outlier.

The foregoing embodiment has described the process of determining the feature matching result, and the process of determining the relative pose will be described in more detail with reference to the drawings and embodiments below.

In some possible embodiments, the determining of the relative pose based on the feature matching result may include estimating an initial relative pose between the query image and the search image based on the feature matching result, based on the initial relative pose. determining a local point corresponding to a key point of the query image in the search image, and forming a point matching pair based on the local point corresponding to the key point; and based on the point matching pair, the relative A step of estimating a pose may be included.

In the implementation process, a local point corresponding to a key point of the query image in the search image may be determined using a projection search matching method, and a point matching pair is formed based on the local point corresponding to the key point. , a relative pose between a query image and a search image may be estimated using a Perspective-n-Point (PNP) algorithm.

In the above embodiment, the detailed process of the relative pose has been described, and after obtaining the relative pose, a global map optimized according to the relative pose can be obtained.

In some possible embodiments, the method executed by the electronic device may further include obtaining an optimized global map based on the relative pose.

In some possible embodiments, an optimized global map may be obtained by combining progressive bundle adjustment (IBA) and full bundle adjustment (FBA), and depending on the relative pose, determining whether to adopt progressive bundle adjustment or full bundle adjustment Optimization accuracy can be improved.

In some possible embodiments, obtaining the optimized global map based on the relative pose may include obtaining the optimized global map by optimizing a current global map based on the relative pose.

In some possible embodiments, during the position estimation and map creation process, the global map is continuously optimized, which may re-optimize the previously optimized global map based on the relative pose. That is, an optimized global map may be obtained by optimizing the current global map.

In some possible embodiments, optimizing a current global map based on a relative pose to obtain an optimized global map includes determining pose drift information based on the relative pose, and the pose drift information and obtaining the optimized global map by determining an optimization strategy based on and optimizing the current global map according to the optimization strategy.

In this case, the pose drift information includes at least one of a drift angle, a drift distance, and the number of closed loops of similar drift.

At this time, the optimization strategy may include incremental bundle adjustment and/or overall bundle adjustment.

Hereinafter, a process of determining pose drift information will be described through some embodiments.

In some possible embodiments, if a loop is successfully detected, the pose drift by the formula below is calculated

in the expression, represents estimating the relative pose between the query image and the search image by the SLAM method, represents the rotational drift, denotes a drift moving in parallel. angle of drift and distance ( )Is and can be calculated by

In some possible embodiments, to determine the accuracy of the relative pose of the closed loop (LC) module, within a time window Closed-loop drift error between the query image of and the current query image ( ) was calculated, and the pose drift was calculated by the equation below.

in the expression, represents the index of the current query image, denotes the drift error of the rotation, denotes the drift error of parallel movement.

In some possible embodiments, the angle of error ( ) and distance ( )Is and can be calculated by Finally, the number of time-matched loops within the time window ( ) to get the statistics. is a predetermined threshold value ( ) or more, the estimated relative position ( ) satisfies the temporal consistency, it means that it is sufficiently accurate.

is represented by the equation below.

in the expression, represents the number of loops whose time matches within the time window.

The process of determining the pose drift information has been described above, and the process of obtaining an optimized global map based on the pose drift information will be described in detail below through some embodiments.

In some possible embodiments, the step of determining an optimization strategy based on the pose drift information and optimizing the current global map according to the optimization strategy to obtain the optimized global map may include determining the pose drift information based on a preset error condition. , obtaining the optimized global map by adjusting the initial global map through gradual bundle adjustment, or if the pose drift information does not meet the error condition, the initial global map through full bundle adjustment. obtaining the optimized global map by adjusting the map.

In other words, if the pose drift information meets the preset error condition, an optimized global map is obtained by adjusting the initial global map by gradual bundle adjustment based on the point matching pair, and the pose drift information meets the preset error condition. If they do not match, an optimized global map is obtained by adjusting the initial global map by adjusting the entire bundle based on the relative pose and point matching pair.

In some possible embodiments, the angle of drift obtained above ( ) and distance ( ) and the number of time-matched loops ( ), the following optimization strategy is executed.

At this time, IBA is incremental bundle adjustment, and FBA is overall bundle adjustment.

If the drift of the camera is very small ( class is a predetermined threshold value ( and ) or less than the estimated relative pose ( ), if the temporal concordance of ( This predetermined threshold ( ) less than), only point matching pair constraints are added, and then the poses and map points of the relevant key frames are optimized through progressive bundle adjustment. Alternatively, if the cumulative error of the current SLAM system is relatively large and the estimated relative pose ( ) satisfies the temporal consistency and is sufficiently accurate, then the estimated relative pose ( ) and point matching pair constraints, and optimize all key frame poses and all map points through full bundle adjustment.

In some possible embodiments, the step of adjusting the initial global map and obtaining an optimized global map through full bundle adjustment may include a Multi-Degree of Freedom pose of a key frame of the initial global map based on the relative pose. Optimizing and acquiring a first global map, and optimizing key frame poses and map points of the first global map by adjusting all bundles, and acquiring an optimized global map.

As shown in FIG. 5, first, the 6 degree of freedom poses of all key frames are optimized, and then all key frame poses and all map points can be optimized through full bundle adjustment (FBA).

Provided in some possible embodiments, a method executed by an electronic device includes acquiring a search image for a query image, determining a relative pose between the query image and the search image, and generating pose drift information based on the relative pose. The method may include determining an optimization strategy based on the pose drift information, and obtaining the optimized global map by optimizing the current global map according to the optimization strategy.

In some possible embodiments, the step of determining the relative pose of the query image and the search image forms a visual constraint between the query image and the search image through feature matching, and estimates the relative pose between the query image and the search image. steps may be included.

In some possible embodiments, feature matching may be BOW and ORB feature matching or projection matching, and the like.

In some possible embodiments, the determining of the relative poses of the query image and the search image may include obtaining a spatial feature of the query image and a spatial feature of the search image, respectively, and the query image and the search image based on the spatial feature. A step of estimating relative poses between search images may be included.

In some possible embodiments, a feature matching result may be obtained by matching a spatial feature of a query image with a spatial feature between a search image at least once, and a relative pose may be estimated again based on the feature matching result.

In an implementation process, multiple layers of coarse-to-fine matching may be performed on the 3D point set of the query image and the 3D point set of the search image, and a relative pose may be determined based on the final matching result. The method for determining the relative pose can be referred to the above description, so it is not further described herein.

In some possible embodiments, when the pose drift information meets a preset error condition, the optimized global map is obtained by adjusting the initial global map through gradual bundle adjustment, and the pose drift information meets the error condition. If they do not match, an optimized global map is obtained by adjusting the initial global map through full bundle coordination.

Hereinafter, a method executed by the electronic device of the present disclosure will be described with reference to embodiments.

In an embodiment, as shown in FIG. 6 , the electronic device of the present disclosure includes an image search module, a query image and a search for a search image semantically similar to a query image in an image data set corresponding to a key frame. An initial relative pose estimation module, which estimates an initial relative pose between images, precisely estimates a relative pose constraint between a query image and a retrieval image, and forms a constraint between a query image key point and a corresponding local map point of the retrieval image. It may include a location estimation module and an optimization module that precisely estimates the optimized global map by further optimizing according to the newly added constraints.

Hereinafter, a method executed by the electronic device of the present disclosure will be further described with reference to embodiments.

As shown in FIG. 7 , a method executed by an electronic device of the present disclosure retrieves an image according to the BoW model, but is semantically similar to a query image (ie, the query image shown in the figure) in an image data set. Retrieving a search image (ie, the search image shown in the figure), generating a 3D point set of the query image, generating a 3D point set of the search image, clustering the 3D point set of the query image Forming at least one first cube, generating at least one second cube by clustering three-dimensional point sets of the search image, determining a first cluster center of each of the first cubes, and determining a first cluster center of each of the second cubes. Determining two cluster centers, determining a second cluster center that matches each of the first cluster centers, and forming the first feature matching pair based on the first and second cluster centers that match each other. As a step, that is, a coarse matching step shown in the figure, forming a second feature matching pair between the 3D point sets of the query image and the search image based on the first feature matching pair, that is, the fine matching step shown in the figure , generating a third feature matching pair between the 3D point sets of the query image and the search image by pose guide matching and generating an initial relative pose; correspondence between the key points of the query image and the search image based on the initial relative pose estimation; determining point matching pairs between local points to be matched and estimating relative poses, and determining an optimization strategy for the initial global map based on the relative poses and point matching pairs, thereby selecting global bundle adjustment or incremental bundle adjustment. Thus, the initial global map obtained through simultaneous localization and map creation can be optimized.

In the foregoing example, a novel closed-loop (LC) method named as a closed-loop (DH-LC) method with hierarchical and mixed properties is presented. The process of 3D point set generation, 3D point set clustering, coarse matching, fine matching, and pose guide matching is named Hierarchical Spatial Feature Matching (HSFM) based on hierarchical segmentation, and optimization combining IBA and FBA The method is named Hybrid Bundle Adjustment (HBA), and the present disclosure optimizes the global map by estimating the initial relative pose and Hybrid Bundle Adjustment (HBA) between the query image and the search image.

For each query image, a search graphic is obtained from a set of candidate images by the BOW model, and then a relative pose between the query image and the search image is estimated in a coarse-to-fine order through HSFM. Next, the matching between the key points of the query image and the corresponding local map points of the search image is completed using the projective search matching method, the accurate relative pose between the query image and the search image is estimated using the PNP algorithm, and finally , according to the proposed optimization strategy, HBA can adaptively choose either the IBA or FBA method to more effectively optimize the current global map.

In order to improve the internal point ratio and efficiency of feature matching, the present disclosure proposes HSFM. Unlike existing methods based on direct local feature matching or feature clustering accelerated matching, the present disclosure first densely and uniformly obtains key points and ORB descriptors from query images and search images, and then performs stereo matching using epipolar constraints. and triangulation are performed to estimate the corresponding 3D points of the query image and the search image, the 3D points are clustered into cubes according to the spatial distribution, and the center descriptor of each cluster is all 3 of the cube with a larger detection viewing angle. It is obtained by voting of dimension point descriptors, and finally, the initial relative pose between the query image key frame and the search image is estimated by a coarse-to-fine method. After robust pose estimation and point matching, the next step is to optimize the global map. Since no single optimization method can consider precision and efficiency at the same time, the present disclosure proposes an HBA that can effectively optimize the global map, has a short execution time, and has higher precision by combining IBA and FBA.

1) HSFM generates 3D points based on coarse-to-fine layer matching and epipolar constraints, and performs spatial clustering to estimate the initial relative pose between the query image and the search image.

Compared with the prior art, the method proposed by the present disclosure improves the internal point ratio and efficiency of feature matching.

2) HBA is a combination of IBA and FBA, effectively providing an optimized global map, with shorter execution time and higher precision.

3) The present disclosure suggests a DH-LC method by combining HSFM and HBA. This method improved the recovery rate and efficiency of the closed loop, reduced the cumulative error, and further improved the accuracy of position estimation.

The method implemented by the above-described electronic device estimates a relative pose between a query image and a search image through spatial features of the query image and search image, the spatial feature includes a larger perceptual aliasing layer and spatial information, and generates a global map. can be more precisely optimized.

In addition, since the present disclosure estimates a set of three-dimensional points by densely and uniformly extracting key points and ORB descriptors from an image, and then completing stereo matching and triangulation using epipolar constraints, the set of three-dimensional points is better than the global map. It is more uniformly and densely distributed in space. This allows a more accurate determination of the relative pose to optimize the global map.

In addition, the present disclosure can effectively optimize the global map through gradual bundle adjustment and overall bundle adjustment, the operation time is shorter, and the accuracy is higher.

In the above embodiment, the method executed by the electronic device is introduced from the viewpoint of the flow of the method, and the present method is introduced from the viewpoint of the virtual module below. Examples include:

In one embodiment, the electronic device 80 provided by the embodiment of the present disclosure includes a first acquisition module 801, a second acquisition module 802, an estimation module 803 and optimization as shown in FIG. It may include a module 804, wherein the first acquiring module 801 is for acquiring a search image for the query image, and the second acquiring module 802 is configured to obtain a spatial feature of the query image and a spatial feature of the search image. for each acquisition, and the estimation module 803 is for estimating an optimized global map of relative poses between the query image and the search image based on the spatial features.

In an embodiment, the spatial feature includes a set of three-dimensional points, the second acquiring module 802 acquires the spatial feature for any one of the query image and the search image, and the second acquiring module 802: It is to extract image feature points including image key points and feature descriptors, and perform stereo matching on the image feature points to estimate a 3D point set.

In one possible embodiment, the estimation module 803 estimates the relative pose between the query image and the search image based on spatial features.

The estimation module 803 is configured to obtain a feature matching result by matching the spatial feature of the query image with the spatial feature of the retrieval image at least once, and determine a relative pose based on the feature matching result.

In an embodiment, the feature matching result includes a first feature matching pair, and the estimation module 803 matches the spatial feature of the query image with the spatial feature of the retrieval image at least once, and obtains a feature matching result.

The estimation module 803 is for clustering each of the three-dimensional point sets of the query image and the search image to generate a first feature matching pair between the clustering result of the query image and the clustering result of the search image.

In one possible embodiment, the estimation module 803 clusters each of the three-dimensional point sets of the query image and the search image to generate a first feature matching pair between the clustering result of the query image and the clustering result of the search image.

The estimation module 803 determines at least one first cube formed by accumulating the three-dimensional point sets of the query image, determines at least one second cube formed by accumulating the three-dimensional point sets of the search image, and determines the first cube formed by accumulating the three-dimensional point sets of the search image. Determining the center of the first cluster of each cube, determining the center of the second cluster of each of the second cubes, determining the center of the second cluster matching each other with the center of the first cluster of each of the first cubes, and determining the center of the first cluster matching each other It is to form the first feature matching pair based on the cluster center and the second cluster center.

In an embodiment, the feature matching result further includes a second feature matching pair, and the estimation module 803 matches the spatial feature of the query image with the spatial feature of the retrieval image at least once to obtain a feature matching result. .

The estimation module 803 performs a nearest neighbor search and mutual verification on 3D points in the region of the first feature matching pair, and performs a second feature matching pair between the 3D point set of the query image and the 3D point set of the search image. is to obtain

In a possible embodiment, the feature matching result further includes a third feature matching pair, and the estimation module 803 matches the spatial feature of the query image with the spatial feature of the search image at least once to obtain a feature matching result. do.

The estimation module 803 estimates an approximate relative pose between the query image and the retrieval image based on the second feature matching pair, and projects a three-dimensional point set of the retrieval image onto a coordinate system of the query image according to the approximate relative pose; This is to determine a third feature matching pair between the 3D point set of the query image and the 3D point set of the search image.

In one possible embodiment, estimation module 803 determines the relative pose based on the feature matching result.

The estimation module 803 estimates an initial relative pose between the query image and the search image based on the feature matching result, and based on the initial relative pose, a local point in the search image corresponding to a key point of the query image. To determine a, form a point matching pair based on the local point corresponding to the key point, and estimate the relative pose based on the point matching pair.

An embodiment further includes an optimization module, wherein the optimization module is configured to optimize the current global map based on the relative pose to obtain an optimized global map.

In one embodiment, the optimization module optimizes the current global map based on the relative pose to obtain an optimized global map.

An optimization module is configured to determine pose drift based on the relative pose, determine an optimization strategy based on the pose drift information, and optimize the current global map according to the optimization strategy to obtain the optimized global map.

In one possible embodiment, an optimization module determines an optimization strategy based on the pose drift information, and optimizes the current global map according to the optimization strategy to obtain the optimized global map.

The optimization module obtains the optimized global map by adjusting the initial global map through gradual bundle adjustment when the pose drift information meets a preset error condition, or the pose drift information does not meet the error condition. If not, it is to obtain the optimized global map by adjusting the initial global map through overall bundle adjustment.

In one possible embodiment, the optimization module obtains the optimized global map by adjusting the initial global map through full bundle coordination.

The optimization module optimizes the multi-degree-of-freedom pose of the key frame of the initial global map based on the relative pose to obtain a first global map, and optimizes the key frame pose and map points of the first global map through adjustment of the entire bundle. and to acquire the optimized global map.

The above-described electronic device estimates a relative pose between a query image and a search image through spatial features of the query image and search image, so that the feature has a larger sensed viewing angle layer and spatial information, and can more accurately optimize a global map.

In addition, since the 3D point set is estimated by densely and uniformly extracting key points and ORB descriptors from the image, and then completing stereo matching and triangulation using epipolar constraints, the 3D point set is It is more uniformly and densely distributed in space than the global map.

In addition, the global map can be effectively optimized through gradual bundle adjustment and overall bundle adjustment, with shorter operating time and higher accuracy.

An electronic device in an embodiment of the present disclosure may execute a method having a similar implementation principle provided by any of the foregoing method embodiments in the present disclosure. Operations executed by each module of the device in each embodiment of the present disclosure correspond to steps in a method executed by an electronic device in each embodiment of the present disclosure, and description of detailed functions of each module of the device can refer to the description of the corresponding method described above, and will not be described further.

An apparatus provided in an embodiment of the present disclosure may implement at least one module among several modules according to an artificial intelligence (AI) model. An apparatus provided in an embodiment of the present disclosure may execute an AI-related function by using a non-volatile memory, a volatile memory, and a processor.

The processor may include one or more processors. At this time, the one or more processors may be, for example, a central processing unit (CPU), a general-purpose processor such as an application processor (AP), or a graphics-only processor, for example, a graphics processing unit (GPU), a visual processing unit (VPU, Visual Processing Unit), and/or an AI dedicated processor such as a Neural Processing Unit (NPU).

The one or more processors control the processing of input data according to non-volatile memory and predefined operating rules or artificial intelligence (AI) models stored in the volatile memory. Predefined operating rules or artificial intelligence models are provided through training or learning.

Here, providing through learning means acquiring an AI model having predefined operating rules or necessary characteristics by applying a learning algorithm to a plurality of learning data. The learning may be executed in the device itself executing the AI according to the embodiment, and/or implemented by a separate server/system.

The AI model may include a plurality of neural network layers. Each layer has multiple weights, and calculation of one layer is performed based on the calculation result of the previous layer and the multiple weights of the current layer. Examples of neural networks include convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), restricted Boltzmann machine (RBM), and deep trust neural network. (DBN, Deep Belief Network), bidirectional recurrent deep neural network (BRDNN), generative adversarial networks (GAN, Generative Adversarial Networks), and deep Q networks.

The learning algorithm is a method of allowing, allowing, or controlling a target device to make a judgment or prediction by training a target device (eg, a robot) set in advance using a plurality of learning data. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

In the above, devices provided by embodiments of the present disclosure are introduced from the viewpoint of functional modularization, and then, electronic devices provided by embodiments of the present disclosure are introduced from the viewpoint of hardware realization, and a calculation system of the electronic device is also introduced.

Based on the same principle as the method shown in the embodiments of the present disclosure, in the embodiments of the present disclosure, an electronic device is additionally provided, and the electronic device performs the operation of the above-described embodiments by calling a memory for storing computer operating instructions and a computer operating command. a processor configured to execute any of the methods, but is not limited thereto. Compared with the prior art, the method implemented by the electronic device of the present disclosure optimizes the global map more precisely.

One embodiment provides an electronic device, and the electronic device 1000 shown in FIG. 9 includes a processor 1001 and a memory 1003. Here, the processor 1001 is connected to the memory 1003 through a bus 1002, for example. Optionally, the electronic device 1000 may further include a transceiver 1004. In actual application, the number of transceivers 1004 is not limited to one, and the structure of the electronic device 1000 does not limit the embodiments of the present disclosure.

The processor 100 includes a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programming support gate array (FPGA, Field Programmable Gate Array) or other editable logic elements, transistor logic elements, hardware components, or any combination thereof. It may implement various illustrative logical frames, modules and circuits described in the present disclosure. The processor 1001 may be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

Bus 1002 may include pathways for conveying information between the components described above. Bus 1002 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, or the like. The bus 1002 can be classified as an address bus, a data bus, a control bus, and the like. For convenience of display, only one thick line is indicated in FIG. 9, but this does not mean that there is only one bus or only one bus.

Memory 1003 may be read only memory (ROM) or other form of static memory device capable of storing static information and instructions, random access memory (RAM) or other form of dynamic memory capable of storing information and instructions. device, electrically erasable programmable read only memory (EEPROM), compact disc read only memory (CD-ROM) or other optical disc memory, compact disc memory (compact disc, laser disc, optical disc, digital universal disc , Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage device, or any other medium accessible to a computer that can be used to carry or store desired program code in the form of instructions or data structures; , but not limited thereto.

The memory 1003 stores application program codes according to an embodiment of the present disclosure, and execution is controlled by the processor 1001 . The processor 1001 is configured to execute application program codes stored in the memory 1003 to implement the contents described in the embodiments of the foregoing method.

Here, the electronic device includes a mobile phone, a laptop computer, a digital broadcasting receiver, a personal digital assistant (PDA), a tablet computer (PAD), a portable multimedia player (PMP), a vehicle-mounted terminal (eg, a vehicle-mounted device) It may include mobile terminals such as navigation terminals), fixed terminals such as digital TVs and desktop computers, and intelligent robots, but is not limited thereto. The electronic device shown in FIG. 9 is only one embodiment and does not limit the function and use range of the embodiment of the present disclosure.

An embodiment of the present disclosure provides a computer readable storage medium. A computer program is stored in the computer-readable storage medium, so that the computer can perform the corresponding contents of the above-described method embodiments when operated by the computer.

Although each step shown in the flowchart of the drawings is shown in the order indicated by the arrow direction, it is needless to say that these steps are not necessarily performed sequentially in the order indicated by the arrow direction. Unless explicitly stated in this disclosure, the execution of these steps is not strictly limited in order and may be performed in other orders. In addition, at least some of the steps in the flowchart of the drawings may include several sub-steps or several procedures, and these steps or procedures do not have to be completed simultaneously and may be executed at different times, and the execution order must also proceed sequentially. However, it may be mixed or alternated with other steps or at least some of its substeps or procedures.

The foregoing computer readable medium of this disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the two.

A computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or element, or any combination thereof. Examples of computer readable storage media are electrical connections having one or more wires, portable computer disks, hard disks, RAM, ROM, erasable ROM (EPROM or flash memory), optical fiber, read-only compact disk storage (CD-ROM) , an optical memory device, a magnetic memory device, or any suitable combination of the foregoing, but is not limited thereto.

In the present disclosure, a computer readable storage medium may be a tangible medium that contains or stores a program, and the program may be used by or in conjunction with an instruction execution system, device, or device.

In this disclosure, a computer readable signal medium may include a data signal propagated in baseband or as part of a carrier wave loaded with computer readable program code. Various forms may be employed as the propagated data signal, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer readable signal medium may be any computer readable medium other than a computer readable storage medium, wherein the computer readable signal medium transmits a program used by or used with an instruction execution system, device, or element; can propagate or transmit. The program code included in the computer readable medium may be delivered using any suitable medium, including, but not limited to, electric wire, optical cable, radio frequency (RF), etc., or any suitable combination thereof. It doesn't.

The computer readable medium may be included in the electronic device, and may exist separately without being assembled in the electronic device.

One or more programs are loaded in the computer readable medium, and when the above-described one or more programs are executed by the electronic device, the electronic device executes the method described in the foregoing embodiment.

Computer program code for carrying out the operations of the present disclosure can be written in one or more programming languages, or combinations thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and the "C" language or similar programming languages. It includes common procedural programming languages such as The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate packet, partially on the user's computer and on the remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of local area network (LAN) or wide area network (WAN), or to an external computer (e.g., by using an Internet service provider to access the Internet). connected by).

The flow diagrams and block diagrams in the drawings illustrate the implementable structures, functions, and operation of systems, methods, and computer programming products according to various embodiments of the present disclosure. In this regard, every block in a flowchart or block diagram may represent a single module, program segment, or portion of code, wherein the module, program segment, or portion of code is one or more executions configured to implement a specified logical function. Contains possible commands. Also, in some alternative implementations, the functions depicted in the blocks may occur in an order different from the order presented in the figures. For example, two blocks displayed as being connected may in fact be basically executed together, and in some cases may be executed in the reverse order, depending on the related functions. In addition, all blocks and combinations of blocks in block diagrams and/or flowcharts may be implemented by dedicated hardware-based systems that perform specified functions or operations, or by a combination of dedicated hardware and computer instructions. It can be.

Modules mentioned in the embodiments of the present disclosure may be implemented in a software or hardware manner. In this case, the name of the module does not limit the module itself in some cases, and for example, the second acquisition module may be described as a “spatial feature acquisition module”.

The above description is only a description of the preferred embodiment of the present disclosure and the technical principles used. The disclosure scope related to the present disclosure is not limited to a technical solution consisting of a specific combination of the foregoing technical features, and other technical solutions arbitrarily combined with the foregoing technical features or equivalent features thereof without departing from the spirit of the present disclosure. include For example, it is a technical solution formed by substituting the above-described features and technical features having similar functions disclosed in the present disclosure (but not limited thereto) to each other.

1000: electronic devices
1001: processor
1003: memory

Claims (14)

As a method performed by an electronic device,
obtaining a search image for the query image;
acquiring spatial features of each of the query image and the search image; and
Estimating a relative pose between the query image and the search image based on the spatial feature; According to claim 1,
The spatial feature comprises a set of three-dimensional points;
The step of obtaining the spatial feature is
extracting image feature points including image key points and feature descriptors; and
Estimating the three-dimensional point set by performing stereo matching on the image feature points. According to claim 2,
The step of estimating the relative pose is
obtaining a feature matching result by matching a spatial feature of the query image with a spatial feature of the search image at least once; and
determining the relative pose based on the feature matching result. According to claim 3,
the feature matching result includes a first feature matching pair;
The step of obtaining the feature matching result is
clustering each of the three-dimensional point sets of the query image and the search image to generate the first feature matching pair between a clustering result of the query image and a clustering result of the search image. According to claim 4,
Generating the first feature matching pair
determining at least one first cube formed by clustering 3D point sets of the query image;
determining at least one second cube formed by clustering 3D point sets of the search image;
determining a first cluster center of each of the first cubes and determining a second cluster center of each of the second cubes; and
Determining a first cluster center of each first cube and a second cluster center matching each other, and determining the first feature matching pair based on the first cluster center and the second cluster center matching each other, method. According to claim 4,
the feature matching result further includes a second feature matching pair;
The step of obtaining the feature matching result is
Obtaining a second feature matching pair between the 3D point set of the query image and the 3D point set of the search image by performing a nearest neighbor search and mutual verification on the 3D points of the first feature matching pair. contains more;
The step of determining the relative pose is
determining the relative pose based on the second feature matching pair. According to claim 6,
the feature matching result further includes a third feature matching pair;
The step of obtaining the feature matching result is
estimating a rough relative pose between the query image and the search image based on the second feature matching pair; Projecting the 3D point set of the search image to the coordinate system of the query image according to the approximate relative pose to determine a third feature matching pair between the 3D point set of the query image and the 3D point set of the search image Step; including,
The step of determining the relative pose is
determining the relative pose based on the third feature matching pair. According to claim 3,
The step of determining the relative pose is
estimating an initial relative pose between the query image and the search image based on the feature matching result;
determining a local point corresponding to a key point of the query image in the search image based on the initial relative pose, and forming a point matching pair based on the local point corresponding to the key point; and
estimating the relative pose based on the point matching pair. According to claim 1,
optimizing a current global map based on the relative pose to obtain an optimized global map. According to claim 9,
Acquiring the optimized global map
determining pose drift information based on the relative pose; and
determining an optimization strategy based on the pose drift information, and obtaining the optimized global map by optimizing the current global map according to the optimization strategy. According to claim 10,
Acquiring the optimized global map
acquiring the optimized global map by adjusting an initial global map through gradual bundle adjustment when the pose drift information meets a preset error condition; or
and acquiring the optimized global map by adjusting an initial global map through full bundle adjustment when the pose drift information does not meet the error condition. According to claim 11,
Acquiring the optimized global map
obtaining a first global map by optimizing a multi-degree-of-freedom pose of a key frame of the initial global map based on the relative pose; and
and obtaining the optimized global map by optimizing key frame poses and map points of the first global map through overall bundle adjustment. In electronic devices,
one or more processors;
Memory;
It includes one or more application programs, the one or more application programs are stored in a memory and configured to be executed by one or more processes, and the one or more application programs are stored in the electronic device according to any one of claims 1 to 12. An electronic device configured to execute an operation corresponding to a method executed by A computer-readable recording medium on which a program for executing the method of claim 1 is recorded on a computer.

Images