CN111445526A - Estimation method and estimation device for pose between image frames and storage medium - Google Patents

Abstract

The present application discloses a method for estimating pose between image frames, an estimating device and a storage medium, specifically, firstly receiving image frames, then using each frame of image frames as a current frame, and using the previous frame of the current frame as a reference frame, and according to the current frame, track the current frame in the reference frame and the local map generated based on the reference frame in turn, and finally, in response to the successful tracking, determine the current frame that meets the preset conditions as the key frame, and extract the image features in the key frame points, and calculate the optimal pose between keyframes based on image feature points. The embodiments of the present application distinguish key frames from non-key frames, and only extract image feature points in key frames, so as to improve the optimization efficiency and optimization accuracy of pose optimization.

Description

Estimation method, estimation device and storage medium for pose between image frames

technical field

The present application relates to the technical field of computer vision, and in particular, to a pose estimation method, estimation device and storage medium between image frames.

Background technique

Simultaneous Localization and Mapping (SLAM) originated in the field of robotics, and generally refers to the technology that robots carry sensors to locate their own position during the movement process, and describe the surrounding environment in an appropriate way. SLAM helps solve the problem of reconstructing the three-dimensional structure of the environment in an unknown environment in real time and positioning the robot itself at the same time, and can present information more efficiently and intuitively than traditional text, image and video methods. When the Global Positioning System (Global Positioning System, GPS) cannot be used normally, SLAM can also be used as an effective alternative to realize real-time navigation in an unknown environment. Therefore, SLAM technology plays an increasingly important role in many fields such as service robots, driverless cars, and augmented reality.

Visual SLAM technology can be divided into feature point extraction method and direct method. Among them, the feature point method extracts salient image features in each image, uses invariant feature descriptions to match feature points in consecutive frames, uses epipolar geometry to robustly restore camera pose and structure, and uses associated features to complete Combined with vision calculation and pose optimization based on minimizing projection error, these extracted salient features can be clustered to describe the entire image for loop closure detection, but the extraction and correlation matching of image feature points are tedious and time-consuming tasks. The direct method can directly recover the pose and structure of the camera through the photometric error without extracting features. However, the direct method does not have the capability of loop closure detection, resulting in large pose drift and accumulated errors in long-term navigation.

SUMMARY OF THE INVENTION

The embodiments of the present application provide a method for estimating pose between image frames, which overcomes the problems of inaccurate and low efficiency of pose estimation between image frames.

The method includes:

receive image frames;

Taking each frame of the image frame as the current frame, and taking the previous frame of the current frame as the reference frame, and according to the current frame, sequentially in the reference frame and the local map generated based on the reference frame track the current frame;

In response to successful tracking, the current frame that satisfies the preset condition is determined as a key frame, image feature points are extracted from the key frame, and an optimal pose between the key frames is calculated based on the image feature points.

Optionally, when the current frame is successfully tracked in the reference frame, the initial poses of the current frame and the reference frame are acquired;

Project the map point tracked in the reference frame to the current frame according to the initial pose, and calculate the image block where the projected point of the map point is projected in the current frame and the corresponding image block in the current frame. The pixel error between the image blocks where the matching point in the current frame is located;

The relative pose after minimizing the pixel error is taken as the optimal pose between the current frame and the reference frame.

Optionally, project the three-dimensional map points in the local map corresponding to at least one of the image frames before the current frame to the current frame, and search and search for the local map in the vicinity of at least one of the projection points. The projection point corresponding to the three-dimensional map point in :

Select the projection point that is closest to the gray value of the matching point in the current frame from the searched projection points, and calculate the relationship between the selected projection point and the corresponding point in the current frame. the pixel error between the matching points;

The relative pose after minimizing the pixel error is taken as the optimal pose between the current frame and the local map.

Optionally, when tracking the current frame in the reference frame fails, extract the image feature point in the current frame that matches at least one map point in the key frame of the previous frame, and calculate relative pose between the current frame and the key frame of the previous frame, and minimize the relative pose to complete pose tracking.

Optionally, when tracking the current frame in the local map fails, extract the image feature point in the current frame that matches at least one map point in the local map, and calculate the current frame. The relative pose between the frame and the local map, and minimize the relative pose to complete the pose tracking.

Optionally, the number of consecutive image frames exceeds a preset number of times and the key frame is not selected in the preset number of consecutive image frames, and/or tracked in the reference frame. The number of map points is less than the preset threshold.

Optionally, extracting the image feature point that matches at least one 3D map point in the local map in at least one of the key frames, and based on the difference between the image feature point and the 3D map point. For the optimal pose after minimizing the pixel error, perform pose optimization on the coordinates of the matched three-dimensional map points;

Compare the similarity between the image feature points of the current key frame and the image feature points of at least one previously determined key frame, and determine a candidate loop closure for the current key frame greater than the similarity threshold frame, and when the number of consecutive similarities between the candidate loopback frame and its adjacent image frames and the previously determined at least one key frame and its adjacent image frames is greater than a preset number, the candidate loopback frame is determined to be Loopback frame.

In another embodiment of the present invention, an apparatus for estimating pose between image frames is provided, the apparatus comprising:

a receiving module for receiving image frames;

The tracking module is configured to take each frame of the image frame as the current frame, and take the previous frame of the current frame as the reference frame, and according to the current frame, in sequence in the reference frame and based on the reference frame tracking the current frame in the generated local map;

The building module is used to, in response to successful tracking, determine that the current frame that satisfies a preset condition is a key frame, extract image feature points in the key frame, and calculate the difference between the key frames based on the image feature points. optimal pose.

In another embodiment of the present invention, a non-transitory computer-readable storage medium is provided, the non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the above-described Steps in a method for estimating pose between image frames.

In another embodiment of the present invention, a terminal device is provided, which includes a processor, and the processor is configured to execute each step in the above-mentioned method for estimating pose between image frames.

Based on the above-mentioned embodiment, the image frame is first received, then each image frame is used as the current frame, and the previous frame of the current frame is used as the reference frame, and according to the current frame, the reference frame and the local map generated based on the reference frame are sequentially generated. At last, in response to successful tracking, the current frame that meets the preset conditions is determined as the key frame, the image feature points are extracted from the key frame, and the optimal pose between the key frames is calculated based on the image feature points. In the embodiment of the present application, by distinguishing key frames and non-key frames, and extracting image feature points only in key frames, the optimization efficiency and optimization accuracy of pose optimization are improved.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present application more clearly, the following drawings will briefly introduce the drawings that need to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore do not It should be regarded as a limitation of the scope, and for those of ordinary skill in the art, other related drawings can also be obtained according to these drawings without any creative effort.

FIG. 1 shows a schematic flowchart of a method for estimating pose between image frames provided by Embodiment 100 of the present application;

2 shows a schematic diagram of the process of using the direct method in the non-critical SLAM algorithm to track and using the feature point extraction method in the SLAM algorithm to perform optimization and closed-loop detection in a key frame provided by Embodiment 200 of the present application;

FIG. 3 shows a schematic diagram of a specific flow of a method for estimating a pose between image frames provided by Embodiment 300 of the present application;

FIG. 4 shows a schematic diagram of an apparatus for estimating pose between image frames provided by Embodiment 400 of the present application;

FIG. 5 shows a schematic diagram of a terminal device provided by Embodiment 500 of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

The terms "first", "second", "third", "fourth", etc. (if present) in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein can, for example, be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having", and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units expressly listed, but may include steps or units not expressly listed or for such process, method, product or Other steps or units inherent in the device.

A complete SLAM framework generally consists of the following four modules: tracking front-end, optimization back-end, loop detection, and map reconstruction. The tracking front-end, that is, the visual odometry, is responsible for initially estimating the motion state between image frames and the position of the road signs; the back-end optimization is responsible for receiving the pose information measured by the visual odometry at different times and calculating the maximum posterior probability estimate; the loopback detection is responsible for judging whether the robot is Return to the original position, and perform loop closure to correct the estimation error; map reconstruction is responsible for constructing a map suitable for the task requirements based on the camera track and image. Based on the problems in the prior art, the embodiments of the present application provide a method for estimating pose between image frames, which is mainly applicable to the technical field of computer vision. By distinguishing key frames and non-key frames, using the feature extraction method in the SLAM algorithm in the key frames, and using the direct method in the non-key frames, the construction accuracy of the 3D map is improved and the construction time is saved. The technical solution of the present invention will be described in detail below with specific embodiments, so as to realize a method for estimating pose between image frames. The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. As shown in FIG. 1 , it is a schematic flowchart of a method for estimating pose between image frames according to Embodiment 100 of the present application. The detailed steps are as follows:

S11, receive an image frame.

In this step, the received image frames are collected by the image collection device. Specifically, the image acquisition device may be a camera, a video camera, or a virtual reality (Virtual Reality, VR) device.

S12 , taking each frame of image frame as the current frame, and taking the previous frame of the current frame as the reference frame, and according to the current frame, sequentially tracking the current frame in the reference frame and the local map generated based on the reference frame.

In this step, for convenience of description, the embodiment of the present application uses each received frame as the current frame, and uses the previous frame of the current frame as the reference frame. After receiving the current frame of each frame, the direct method in the SLAM algorithm is used to first track the current frame in the reference frame, and after the tracking is successful, it continues to track the current frame in the local map generated based on the reference frame. Specifically, the process of tracking the current frame in the reference frame and the local map generated based on the reference frame is completed by the tracking thread. Wherein, if the tracking of the reference frame is successful, a constant velocity motion model is used to predict the initial pose of the current image acquisition device. Further, map points tracked in the reference frame are projected onto the current frame based on the estimated initial pose. Finally, the relative pose between the reference frame and the current frame is solved by minimizing the photometric error between image patches corresponding to the same map point.

Further, the relative pose of the current frame obtained only by using the reference frame may be inaccurate. The tracking thread tracks the current frame in the local map for more matching map points that should be tracked and optimized. Among them, the local map is formed by initializing multiple reference frames. Specifically, the relative pose of the pixels in the current frame is matched with the 3D map points in the local map by tracking the relative pose of the current frame in the reference frame, and the 3D map points in the local map corresponding to the previous image frame are projected to the current frame. and select the projection point in the projection area that is closest to the gray value of the matching point on the current frame. By minimizing the photometric error between the projection point and the current matching point, the corresponding feature positions in the current frame are obtained respectively, so that the relative pose of the image acquisition device corresponding to the current frame can be further optimized.

S13, in response to the successful tracking, determine the current frame that meets the preset condition as a key frame, extract image feature points in the key frame, and calculate the optimal pose between the key frames based on the image feature points.

In this step, when the current frame is successfully tracked in the reference frame and the local map generated based on the reference frame in turn, it is further determined whether the current frame is a key frame satisfying the preset condition. Among the preset conditions, the embodiments of the present application include but are not limited to the following four types; a key frame has not been selected for a continuous preset number of times, such as 20 frames; the local optimization thread is in an idle state; the number of tracked map points is less than a preset value such as 50 ; The number of image feature points tracked to the previous key frame in the current frame is less than a preset threshold such as 90% of its total number. Use the feature point extraction method in the SLAM algorithm to extract the Oriented FASTand Rotated BRIEF(ORB) ORB only for the key frames as the image feature points, and calculate the optimal pose between the key frames based on the image feature points. After the current frame is selected as a key frame, it will be passed to the local optimization thread to generate new map points and participate in map point pose optimization, and passed to the loop loop detection thread for loop detection to complete the construction of the 3D map.

As described above, based on the above-mentioned embodiment, the image frame is first received, then each frame of image frame is used as the current frame, and the previous frame of the current frame is used as the reference frame, and according to the current frame, the reference frame and the reference frame based on the current frame are used in turn. The current frame is tracked in the generated local map, and finally, in response to the successful tracking, the current frame that meets the preset conditions is determined as the key frame, the image feature points are extracted from the key frame, and the optimal value between the key frames is calculated based on the image feature points. pose. In the embodiment of the present application, by distinguishing key frames and non-key frames, and extracting image feature points only in key frames, the optimization efficiency and optimization accuracy of pose optimization are improved.

A method for estimating pose between image frames in the embodiment of the present application is mainly a semi-direct SLAM system with loop closure detection function, which can achieve accuracy comparable to that of the feature point method, and enables the system to achieve as much accuracy as possible. Run fast. As shown in FIG. 2 , it is a schematic diagram of the process of using the direct method in the SLAM algorithm for tracking in the non-critical and using the feature point extraction method in the SLAM algorithm in the key frame to perform optimization and closed-loop detection shown in Embodiment 200 of the present application. . Among them, whether the current frame is a key frame is determined according to the degree of change of the scene, the image point features are extracted in the key frame, and the feature point extraction method is used to complete the relative pose calculation between the key frames, and the direct method is used to track in the non-key frames. These features complete fast positioning. Among them, the extracted features are Oriented FAST and RotatedBRIEF (ORB) features. This feature selects Oriented FAST corner points as image feature point coordinates and is described by a 256-dimensional Rotated BRIEF descriptor. This feature has good scale and perspective invariance , the speed of feature extraction and matching is fast, and it is more suitable for real-time computing SLAM systems. The method for estimating pose between image frames shown in the embodiments of the present application includes three threads running in parallel, namely front-end tracking, local optimization, and loop closure detection. The tracking thread is responsible for initially solving the pose of the image acquisition device in each image frame and determining whether to insert a new key frame; the local optimization thread only processes the new key frame, re-optimizes the pose of each frame, and manages the local map application. in pose tracking. The loopback detection thread calculates the similarity between the current keyframe and the previous keyframe to detect loopbacks, and corrects drift errors when a loopback is detected to maintain global consistency.

As shown in FIG. 3 , it is a schematic diagram of a specific flow of a method for estimating a pose between image frames provided by Embodiment 300 of the present application. The method shown in the embodiment of the present application mainly includes four parts: monocular initialization, initial positioning according to reference frame, positioning according to local map, and management of map and image frames. Fig. 3 describes the whole process of the tracking thread after the initialization is completed. After receiving the image frame, it firstly uses the direct method to track the reference frame to determine the pose T _i of the current frame (i is an integer greater than or equal to 1, indicating the i-th current frame frame), if the tracking is successful, continue to use the direct method to track the local map to fine-tune the relative pose of the current frame, if the tracking fails, extract the image feature points of the current frame and use the feature point extraction method to track the previous key frame T _k-1 ( k is an integer greater than or equal to 1, indicating the order of the current key frame) and then track the local map. If tracking the local map fails, extract the feature points and use the feature extraction point method to track the local map, and then update the variables such as speed and local map. Then it is determined whether the current frame is a key frame, and if it is a key frame, the image features are extracted and passed to the local optimization thread. In summary, the detailed process of the specific process is as follows:

S301, receiving an image frame.

Here, each image frame is taken as the current frame, and the previous frame of the current frame is taken as the reference frame.

S302, initialize and acquire the initial poses of the current frame and the reference frame, and determine the local map.

Here, the goal of monocular initialization is to compute the relative pose between two image frames and triangulate a set of 3D map points for tracking in the next image frame. In the initialization process, ORB features are extracted for each frame, and two geometric models are calculated in parallel: the homography matrix H under the assumption of flat scene and the fundamental matrix F under the assumption of non-planar scene, and then use a heuristic method to select one of the models, and use The method corresponding to this model solves the initial pose, and the specific steps are as follows:

(1) Select the initialization frame: In the received image frame, select two consecutive image frames with matching ORB feature points greater than 100 as the initialization frame, which can avoid initialization in the case of low texture and poor illumination.

(2) Two models are calculated in parallel: the basic matrix F is solved by the direct linear transformation of the homography matrix H and the eight-point method respectively, and the above two matrices are solved by Random Sample Consensus (RANSAC) with the same number of iterations. .

(3) Solving model selection: If the scene is flat, close to flat or with low parallax, it can be explained by the homography H. At the same time, the fundamental matrix F can also be solved, but the model will not be well constrained, and there is a large error in the motion estimated from this fundamental matrix. On the contrary, in a non-planar scene, the fundamental matrix F should be selected to estimate the motion, and the evaluation criteria are as follows: Equation 1:

If R _H > 0.45, choose the homography matrix, otherwise, choose the fundamental matrix.

(4) Relative pose and structure solution: When solving the relative pose, if no one model is significantly better than the other, the initialization is not performed, and the process returns to step (1) to start again. This operation is well guaranteed. The robustness of the scene at initialization time. When selecting the basic matrix model, it is necessary to continue to solve the essential matrix E according to the camera internal parameter K, as shown in the following formula 2:

E=K ^T FK Equation 2

And E=t^R, according to the essential matrix E, the solution of four sets of pose transformation can be solved, and then the correct solution can be selected according to the positive and negative values of the depth of the map point in the camera.

(5) Global Bundle Adjustment: Then perform full BA (Global Bundle Adjustmen, the pose and map structure of the image acquisition device participate in the optimization at the same time) to fine-tune the image acquisition device and map structure.

The above are the steps of monocular initialization. After the initialization is completed, the relative pose of the initial two frames is determined, and by initializing some image frames (such as the ten image frames at the beginning of the initialization), a local map that can be used for subsequent frame tracking is generated.

S303, track the current frame in the reference frame.

Here, according to the current frame, the tracking thread sequentially tracks the current frame in the reference frame and the local map generated based on the reference frame.

S304, calculate the optimal pose of the current frame and the reference frame.

Here, when the current frame is successfully tracked in the reference frame, the initial poses of the current frame and the reference frame are obtained. Specifically, after the successful initialization or the successful tracking of the reference frame _, the initial value T _i,i-1 = T _i-1,i-2 , since the current frame does not extract image feature points, based on this initial value, the photometric error is used to optimize the pose between adjacent frames:

u是第i-1帧图像中角点像素位置,R是采集图像角点的区域，T_i,i-1∈SE(3)是由李群表示的6自由度位姿，δI是光度误差。du是第i-1帧中像素u所对应的深度值。in,

u is the pixel position of the corner point in the i-1th frame image, R is the area where the corner points of the image are collected, T _i,i-1 ∈ SE(3) is the 6-DOF pose represented by the Lie group, and δI is the photometric error . du is the depth value corresponding to pixel u in frame i-1.

Further, the map point tracked in the reference frame is projected to the current frame according to the initial pose, and the image block where the projected point of the map point is projected in the current frame and the image corresponding to the matching point in the current frame are calculated. Pixel error between blocks. The relative pose after minimizing the pixel error is taken as the optimal pose between the current frame and the reference frame. Among them, the patch corresponding to each pixel point in the optimization process can contain 8 pixels, of which the pixel in the lower right corner is omitted, and 8 pixels are convenient to start the SSE acceleration calculation in the processor. This patch model is between speed and accuracy. A good balance is achieved.

S305, when the tracking fails, extract image feature points in the current frame to complete the pose tracking of the current frame in the reference frame.

Here, when tracking the current frame in the reference frame fails, extract image feature points in the current frame that match at least one map point in the key frame of the previous frame, and calculate the relative relationship between the current frame and the key frame of the previous frame. pose, and minimize the relative pose to complete the pose tracking. Specifically, after the matching is completed, the pose tracking is completed by minimizing the relative pose between the current frame and the previous key frame (k-1), as shown in Equation 4;

Among them, p _j is the three-dimensional coordinate of the jth image feature point extracted in the key frame k-1 in the camera coordinate system of this frame, u' _j is the image feature point extracted in the current frame and the jth image in the key frame Points that match the feature points.

S306, perform pose optimization on the current frame.

S307, track the current frame in the local map.

In this step, after completing the pose estimation of the current frame based on the reference frame, the local map is continued to be tracked. The local map maintains the 3D map points corresponding to multiple key frames. Constraints can be further increased by tracking the 3D map points of more frames. , Improve the accuracy of pose estimation.

S308, calculate the optimal pose of the current frame and the local map.

Here, the three-dimensional map points in the local map corresponding to at least one image frame before the current frame are projected to the current frame, and the projection points corresponding to the three-dimensional map points in the local map are respectively searched near at least one projected point: Select the projection point that is closest to the gray value of the matching point in the current frame from the obtained projection points, and calculate the pixel error between the selected projection point and the matching point in the current frame; after minimizing the pixel error The relative pose of is the optimal pose between the current frame and the local map. Specifically, the local map point is projected to the current frame to perform fine-tuning of the pose again through the photometric error, as shown in Equation 5:

Among them, p _{j, k-1} is the three-dimensional map point in the world coordinate system, u _j is the pixel from which p _j is extracted, and k is the key frame where u _j is located. Further, it is necessary to further improve the pose accuracy through pixel block matching and projection error optimization. Since the image feature points are not extracted in the current frame, the pixel patch matching needs to be completed through the pixel error of the patch. Taking the fine-tuned pose as the initial value, project the local map points to the current frame, and search for matching patches in the local map points near the projected points. After the matching is completed, the reprojection error model is used to further optimize the pose, as shown in Equation 6:

Among them, u' _j is the searched matching point of p _j .

S309, when the tracking fails, extract image feature points in the current frame to complete the pose tracking of the current frame in the local map.

Here, when tracking the current frame in the local map fails, extract image feature points in the current frame that match at least one map point in the local map, calculate the relative pose between the current frame and the local map, and minimize Relative pose to complete pose tracking. Among them, if the tracking of the local map fails, the ORB feature is extracted in the current frame to complete the matching with the local map, and then the pose is optimized according to formula 6. A cache mechanism is used in the management of local maps. The map points with better tracking effect in the previous frame will be cached and projected to the current frame first. If the cached map points are not enough, other map points will be added. This enables tracking of local maps with greater efficiency.

S310: Determine whether the current frame satisfies the preset condition of the key frame.

Here, determining whether the current frame is selected as a key frame is mainly determined according to the magnitude of the motion range and the scene at that time. The main standard refers to ORB-SLAM. If one of the following conditions is met, the current frame is selected as a key frame. When the number of consecutive image frames exceeds a preset number of times and no key frames are selected in a preset number of consecutive image frames, such as 20 frames, and/or the number of map points tracked in the reference frame is less than a preset threshold such as 90%. It also includes that the local optimization thread is in an idle state and/or the number of map points tracked is less than a preset value such as 50. Meanwhile, when it is not a key frame, the above steps S301 to S310 are continued to be repeated.

S311, perform local optimization of the three-dimensional map based on the key frame and the local map.

Here, an image feature point matching at least one 3D map point in the local map is extracted in at least one key frame, and based on the optimal pose between the image feature point and the 3D map point after minimizing the pixel error, for The coordinates of the matched 3D map points are optimized for pose. Specifically, the main function of the local optimization thread is to perform local Bundle Adjustment (Local BA) on the nearest m key frames and the corresponding local map. The pose of the key frame and the three-dimensional coordinates of the map point are optimized as optimization variables at the same time, and the pose and map accuracy are improved at the same time, and unstable map points and key frames are eliminated according to the tracking and visibility of the map points in the key frame. And triangulate the new map point.

S312, perform loop closure detection of the three-dimensional map.

Here, the similarity is compared between the image feature point of the current key frame and the image feature point of at least one key frame that has been determined before, and the current key frame greater than the similarity threshold is determined as a candidate loop closure frame, and the candidate loop closure frame is When the number of consecutive similarities between the frame and its adjacent image frames and the previously determined at least one key frame and its adjacent image frames is greater than a preset number, the candidate loop closure frame is determined to be a loop closure frame.

Specifically, the loop closure detection thread mainly completes similar key frame judgment, approximate transformation, loop closure fusion, and essential map optimization, as follows:

(1) Detect loop closure: Calculate the similarity between images by the Bag-of-Visual-Words method, first calculate the similarity between the current key frame and its co-view key frame (more than 30 co-view map points), And save the minimum value among them. Frames with a similarity greater than this value are determined as candidate loopback frames, and the current frame and the co-view key frame are matched with the candidate loopback frame and its adjacent frames. If the number of consecutive similar frames is greater than the preset value If 3, it is determined that the candidate frame is a loopback frame.

(2) Calculate the similarity transformation: Since ORB matching can be performed between the current key frame and the loopback frame, a matching relationship is also established between the respective map points, so the similarity transformation between the two key frames can be optimized.

(3) Loop fusion: After completing the solution of the similar pose transformation, adjust the position and attitude of the current key frame and surrounding key frames, so that the two ends of the loop are basically aligned, and then fuse the current key frame and the matching map between the loop frames .

(4) Loop closure: optimize the position and attitude of all key frames in the loop according to the essential graph (the graph formed by establishing edges between the viewpoints in the loop and more than 100 key frames), and evenly distribute the loop error to the corresponding key frames.

The above are the main steps of the loopback detection thread. Since the ORB feature points are extracted from the key frame, SVL has the function of loopback detection.

The present application implements the above-mentioned method for estimating pose between image frames based on the above steps. By extracting image point features in key frames, and completing feature matching between key frames according to the descriptors of the features; non-key frames no longer extract and match image point features, and complete pose tracking and positioning through sparse image alignment . The time consumption caused by feature extraction and matching is avoided in non-key frames, and the ORB features at key frames enable the scheme to have high accuracy and loop closure detection capabilities.

Based on the same inventive concept, Embodiment 400 of the present application further provides a device for constructing a three-dimensional map, wherein, as shown in FIG. 4 , the device includes:

a receiving module 41 for receiving image frames;

The tracking module 42 is used to use each frame of image frame as the current frame, and the previous frame of the current frame as the reference frame, and according to the current frame, in turn in the reference frame and the local map generated based on the reference frame. Track the current frame;

The building module 43 is configured to, in response to successful tracking, determine the current frame satisfying the preset condition as a key frame, extract image feature points in the key frame, and calculate the optimal pose between the key frames based on the image feature points.

In this embodiment, for the specific functions and interaction modes of the receiving module 41 , the tracking module 42 and the building module 43 , reference may be made to the description of the embodiment corresponding to FIG. 1 , which will not be repeated here.

As shown in FIG. 5 , another embodiment 500 of the present application further provides a terminal device, including a processor 501, wherein the processor 501 is configured to execute the steps of the above-mentioned method for estimating a pose between image frames. It can also be seen from FIG. 5 that the terminal device provided in the above-mentioned embodiment further includes a non-transitory computer-readable storage medium 502 , and a computer program is stored on the non-transitory computer-readable storage medium 502 , and the computer program is executed by the processor 501 When performing the steps of the above-mentioned method for estimating pose between image frames. In practical applications, the terminal device may be one or more computers, as long as it includes the above-mentioned computer-readable medium and a processor.

Specifically, the storage medium can be a general-purpose storage medium, such as a removable disk, a hard disk, and FLASH, etc. When the computer program on the storage medium is run, it can execute the above-mentioned method for estimating the pose between image frames. each step. In practical applications, the computer-readable medium may be included in the device/apparatus/system described in the above embodiments, or may exist alone without being assembled into the apparatus/apparatus/system. The above-mentioned computer-readable storage medium carries one or more programs, and when the above-mentioned one or more programs are executed, each step in the above-mentioned method for estimating a pose between image frames can be performed.

According to the embodiments disclosed in the present application, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as, but not limited to, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM) ), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above, but are not intended to limit this application scope of protection. In the embodiments disclosed in this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

The flowchart and block diagrams in the figures of the present application illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments disclosed herein. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more functions for implementing the specified logical function(s) executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the different figures. For example, two blocks shown in connection may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams or flowchart illustrations, and combinations of blocks in the block diagrams or flowchart illustrations, can be implemented in special purpose hardware-based systems that perform the specified functions or operations, or can be implemented using A combination of dedicated hardware and computer instructions is implemented.

Those skilled in the art will appreciate that various combinations and/or combinations of features recited in various embodiments of the present disclosure and/or claims are possible, even if such combinations or combinations are not explicitly described in this application. In particular, various combinations and/or combinations of the features recited in the various embodiments of the present application and/or the claims may be made without departing from the spirit and teachings of the present application, all such combinations and/or combinations falling within Scope of this disclosure.

Finally, it should be noted that the above-mentioned embodiments are only specific implementations of the present application, and are used to illustrate the technical solutions of the present application, rather than limit them. The embodiment describes the application in detail. Those of ordinary skill in the art should understand that: any person skilled in the art can still make changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the application. Or can easily think of changes, or equivalently replace some of the technical features; and these changes, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions in the embodiments of the present application, and should be covered in the present application. within the scope of protection. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims (10)

1. A method for estimating a pose between image frames, comprising:

receiving an image frame;

taking each frame of the image frame as a current frame, taking a previous frame of the current frame as a reference frame, and sequentially tracking the current frame in the reference frame and a local map generated based on the reference frame according to the current frame;

and responding to the successful tracking, determining the current frame meeting the preset conditions as a key frame, extracting image feature points from the key frame, and calculating the optimal pose between the key frames based on the image feature points.

2. The estimation method according to claim 1, wherein between the step of tracking the current frame in the reference frame and the local map generated based on the reference frame in turn and the step of determining that the current frame satisfying a preset condition is a key frame, the method further comprises:

when the current frame is successfully tracked in the reference frame, acquiring initial poses of the current frame and the reference frame;

projecting the map points tracked in the reference frame to the current frame according to the initial pose, and calculating pixel errors between image blocks where the map points are projected in the current frame and image blocks where the matching points corresponding to the current frame are located;

and taking the relative pose after the pixel error is minimized as the optimal pose between the current frame and the reference frame.

3. The estimation method according to claim 2, wherein after the step of taking the relative pose after minimizing the pixel error as the optimal pose between the current frame and the reference frame, the method further comprises:

projecting three-dimensional map points, corresponding to a local map, of at least one image frame before the current frame to the current frame, and searching for the projection points corresponding to the three-dimensional map points in the local map near the at least one projection point respectively:

selecting the projection point which is closest to the gray value of the matching point in the current frame from the searched projection points, and calculating the pixel error between the selected projection point and the matching point corresponding to the current frame;

and taking the relative pose after the pixel error is minimized as the optimal pose between the current frame and the local map.

4. The estimation method according to claim 2, wherein between the step of tracking the current frame in the reference frame and the local map generated based on the reference frame in turn and the step of determining that the current frame satisfying a preset condition is a key frame, the method further comprises:

when tracking the current frame in the reference frame fails, extracting the image feature point matched with at least one map point in the key frame of the previous frame in the current frame, calculating the relative pose between the current frame and the key frame of the previous frame, and minimizing the relative pose to complete pose tracking.

5. The estimation method according to claim 3, characterized in that after the step of taking the relative pose after minimizing the pixel error as the optimal pose between the current frame and the reference frame, the method further comprises:

when tracking of the current frame in the local map fails, extracting the image feature point matched with at least one map point in the local map from the current frame, calculating a relative pose between the current frame and the local map, and minimizing the relative pose to complete pose tracking.

6. The estimation method according to claim 1, wherein the step of determining the current frame satisfying a preset condition as a key frame comprises:

the number of the continuous image frames exceeds a preset number of times, the key frames are not selected from the continuous image frames of the preset number of times, and/or the number of the map points tracked in the reference frame is less than a preset threshold value.

7. The estimation method according to claim 3, wherein after the step of calculating optimal poses between the keyframes based on the image feature points, the method further comprises:

extracting the image feature points matched with at least one three-dimensional map point in the local map from at least one key frame, and performing pose optimization on the coordinates of the matched three-dimensional map points on the basis of the optimal pose between the image feature points and the three-dimensional map points after the pixel error is minimized;

comparing the image feature points of the current key frame with the image feature points of at least one key frame which is determined before, determining a candidate loop-back frame from the current key frame which is greater than a similarity threshold, and determining the candidate loop-back frame as a loop-back frame when the number of the candidate loop-back frame and the adjacent image frame thereof which are continuously similar to the at least one key frame and the adjacent image frame thereof which are determined before is greater than a preset number.

8. An estimation apparatus of a three-dimensional map, characterized in that the construction apparatus comprises:

a receiving module for receiving the image frame;

the tracking module is used for taking the image frame of each frame as a current frame, taking the previous frame of the current frame as a reference frame, and sequentially tracking the current frame in the reference frame and a local map generated based on the reference frame according to the current frame;

and the construction module is used for responding to the successful tracking, determining the current frame meeting the preset conditions as a key frame, extracting image feature points from the key frame, and calculating the optimal pose between the key frames based on the image feature points.

9. A non-transitory computer readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the steps of a method of estimating pose between image frames as claimed in any one of claims 1 to 7.

10. A terminal device characterized by comprising a processor for executing each step in a method of estimating a pose between image frames according to any one of claims 1 to 7.

Images