KR20220122471A - Method and apparatus for simultaneous localization and mapping (SLAM), computer-readable recording medium - Google Patents

Abstract

A simultaneous localization and mapping method, apparatus, and a computer readable medium include: obtaining a structure of a current image frame; obtaining structure re-identification constraints based on the structure of the current image frame, and obtaining a SLAM result based on a structure re-identification constraint. By introducing the structure re-identification constraint in a bundle adjustment process for obtaining the SLAM result, error accumulation can be better suppressed, and the accuracy and robustness in obtaining the SLAM result can be improved.

Description

Simultaneous localization and mapping (SLAM) method and apparatus, and computer-readable storage medium

FIELD OF THE INVENTION The field of localization technology relates, and more particularly, to a simultaneous localization and mapping (SLAM) method, apparatus, and computer-readable storage medium.

When using sensors such as cameras and inertial measurement devices on the device, simultaneous localization and map creation of a technology that constructs a three-dimensional map of the space where the device is located and determines the location and pose of the device on the map in real time (Simultaneous Localization And Mapping, SLAM). Since cameras and inertial measurement devices are relatively inexpensive compared to LiDAR (light detection and ranging) sensors, they are standard components of devices such as mobile phones, augmented reality glasses, and indoor robots, and can be used in various situations. The main research content of all existing SLAM technologies has been on how to map and acquire poses of devices in real time using cameras and inertial measuring devices as sensors. Compared to a monocular camera, a three-dimensional map created by a binocular camera has a real and physical scale, so in practical applications, the visual sensor of the device is usually a binocular camera.

The existing vision-based SLAM system is mainly based on multi-view geometry theory, and uses tracking matching for feature points and feature lines of the image to determine the pose of the device (here, the 'pose of the device' refers to the three-dimensional spatial position of the device and the direction) and 3D environment information. However, the existing SLAM system does not suppress the trajectory drift (ie, accumulated error) well, and thus the robustness in obtaining the SLAM result is also relatively insufficient.

It provides a method and apparatus for simultaneous localization and mapping, and a computer-readable storage medium, and other technical problems can be inferred from the embodiments described below.

In one aspect, a simultaneous localization and mapping (SLAM) method provided by this embodiment includes: obtaining a structure of a current image frame; obtaining a structure re-identification constraint based on the structure of the current image frame; and obtaining a SLAM result based on the structure re-identification constraint.

In this embodiment, the method includes: obtaining points and lines of the current image frame; Re-identifying the points and lines of the current image frame to obtain a point-line re-identification constraint; further comprising, wherein the step of obtaining a SLAM result based on the structure re-identification constraint includes: and obtaining a SLAM result based on the identification constraint.

In this embodiment, obtaining the current image frame structure includes: obtaining an initial structure based on a line of the current image frame; and optimizing the initial structure based on the lines of the current image frame to obtain the structure.

In this embodiment, obtaining the structure re-identification constraint based on the structure of the current image frame includes: obtaining a global sub-map based on the current image frame and the key image frame; and re-identifying the structure based on the global sub-map and obtaining the structure re-identification constraint.

In this embodiment, based on the current image frame and the key image frame, obtaining a corresponding global submap includes: a time sequence relationship between the current image frame and the key image frame, and a three-dimensional (3D) corresponding current image frame and obtaining an entire sub-map based on the spatial relationship between the map and the key image frame and the corresponding three-dimensional map.

In this embodiment, obtaining a global sub-map based on a time sequence relationship between the current image frame and the key image frame, and a spatial relationship between a three-dimensional map corresponding to the current image frame and a three-dimensional map corresponding to the key image frame is: obtaining a reference key image frame of the current image frame based on a time sequence relationship between the current image frame and the key image frame; Based on the spatial relationship between the current image frame and the corresponding three-dimensional map and the key image frame and the corresponding three-dimensional map, the key image frame having an overlapping area with the reference key image frame in the spatial distribution is the spatial common view key of the current image frame. setting as an image frame; and obtaining a global sub-map based on the key image frame having a spatial common view.

In the present embodiment, the reference key image frame is the key image frame closest to the current image frame in time sequence among the key image frames.

In this embodiment, the step of re-identifying the structure based on the global sub-map includes: obtaining a common spatial view key image frame of the current image frame based on the global sub-map; and re-identifying the structure based on the common view correlation structure of the common spatial view key image frame.

In the present embodiment, the step of re-identifying the structure based on the spatial common view correlation structure of the spatial common view key image frame includes: the temporal structure of the current image frame and the image frame within the corresponding first preset time sequence sliding window. based on the correlation structure, perform a temporal consistency test on the structure; If the temporal consistency test for the structure is passed, the temporal correlation structure of the image frame within the sliding window corresponding to the second preset time sequence corresponding to the current image frame, and the third preset time sequence corresponding to the key image frame having a spatial common view match the structure based on the spatial common view correlation structure of the image frame within the sliding window; After the matching is implemented, a temporal-spatial consistency test is performed on the structure based on the temporal correlation structure and the key image frame having the corresponding spatial common view.

In the present embodiment, the step of performing a temporal consistency test on the structure based on the structure of the current image frame and the temporal correlation structure of the image frame in the first preset time sequence sliding window corresponding thereto includes: the structure of the current image frame and the obtaining a relative rotation error between temporal correlation structures of image frames within a sliding window corresponding to a first preset time sequence; and performing a temporal consistency test on the structure based on all relative rotation errors corresponding to the first preset time sequence sliding window.

In the present embodiment, the step of performing the temporal-spatial consistency test based on the temporal correlation structure and the corresponding spatial common view key image frame includes: a relative rotation drift error between the temporal correlation structure and the corresponding spatial common view key image frame. obtaining a; obtaining a relative rotation drift error corresponding to the second preset time sequence sliding window and the third preset time sequence sliding window; and performing a time-spatial consistency test on the structure based on a relative rotation drift error corresponding to the second preset time sequence sliding window and the third preset time sequence sliding window.

In this embodiment, the current image frame is a key image frame, and the step of obtaining a SLAM result based on the dot-line re-identification constraint and the structure re-identification constraint is: , obtaining a global bundle adjusted SLAM result; Based on the dot-line re-identification constraint of all image frames in the fourth preset time sequence sliding window corresponding to the current image frame, performing local bundle adjustment on the entire bundle adjusted SLAM result to obtain a SLAM result. include

In another aspect, the present embodiment provides a structure obtaining module used to obtain a structure of a current image frame; a structure re-identification constraint obtaining module used to obtain structure re-identification constraints based on the structure of the current image frame; and a SLAM result acquisition module used to acquire a SLAM result based on the structural re-identification constraint.

In another aspect, the embodiment includes a memory and a processor; a computer program is stored in the memory; An electronic device is provided, wherein the processor executes a computer program to implement the method provided in the embodiments of the first aspect or any selectable embodiments of the first aspect.

In another aspect, the present embodiment provides a computer-readable storage medium having stored thereon a computer program that, when executed by a processor, implements the method provided in the embodiment of the first aspect or a selectable embodiment of the first aspect.

According to the above, by introducing the structure re-identification constraint in the bundle adjustment process for obtaining the SLAM result, it is possible to better control the accumulated error and improve the accuracy and robustness in obtaining the SLAM result.

1 is a structural diagram of a simultaneous localization and mapping (SLAM) system provided by an embodiment.
2 is a flowchart of a SLAM method provided by an embodiment.
3 is a detailed flowchart of a SLAM method in an example of an embodiment.
4 is a diagram of a relationship between an image plane, an equivalence plane, and the Manhattan world in one example of an embodiment.
5 is a diagram illustrating a case in which three vanishing directions orthogonal to each other are finely estimated in an example of one embodiment.
6 is a structural diagram of a spatiotemporal sensitive global sub-map in one example of an embodiment.
7 is a flowchart for pose guide validation and matching of a Manhattan frame in one example of an embodiment.
8 is a temporal consistency test diagram of a Manhattan frame in an example of an embodiment.
9 is a temporal-spatial consistency test diagram of a Manhattan frame in one example of an embodiment.
10 is a diagram illustrating a SLAM method in an example of an embodiment.
11 is a block diagram illustrating a structure of a SLAM device provided according to an embodiment.
12 is a structural diagram of an electronic device provided according to an embodiment.

Hereinafter, this embodiment will be described in detail. Examples of the embodiment are shown in the drawings, wherein the same or similar reference numerals consistently indicate the same or similar parts or parts having the same or similar functions. Hereinafter, the embodiments described with reference to the drawings are exemplary, and are used only to describe the present embodiment, and should not be construed as limiting the present embodiment.

It will be understood by those skilled in the art that the singular forms "1", "an", "the", or "there" used in this text may include the plural forms, unless specifically stated otherwise. Also, as used herein, the word "comprises" refers to the presence of said feature, integer, step, action, part and/or component, but one or more other features, integer, step, action, part, component. It will also be understood that this does not exclude the possibility of the presence or addition of and/or combinations thereof. It should also be understood that when a component is referred to as “connected” or “coupled” to another component, it may be directly connected or coupled to the other component, although other components may exist in between. In addition, "connection" or "coupling" as used herein may include wireless connection or wireless coupling. As used herein, the word “and/or” includes all or any one unit and combination of one or more of the listed items in relation to each other.

An embodiment of the present embodiment will be described in more detail with reference to the drawings so that the purpose, technical solution, and advantage of the present embodiment are more clear.

In the case of the existing SLAM system, specifically, the point and line features of frequencies in the images related to each other in time sequence can be tracked and matched according to the multi-view geometry principle, and the point and line features of the binocular image are epipolar constrained. (epipolar constraint), and finally, this matching can construct the geometric constraint relationship of the device pose and three-dimensional (3D) map, and can be matched with the device pose through the method of smoothing or bundle adjustment. You can get a 3D map.

Although the closed-loop method that is currently widely used can reduce the accumulated error, the recovery rate is relatively low and is effective only when a closed-loop is found, so that the historical information cannot be sufficiently used to suppress the trajectory drift. On the other hand, the majority of existing methods focus on general situations, and relatively few focus on special situations such as artificial environments. These environments have very strong structural rules, and many of them are abstracted into one box world (Manhattan world) or multiple box worlds (Atlanta world) on a horizontal plane. In the Manhattan world environment, a vertical plane or line occupies a dominant position. These properties have already been applied to indoor 3D reconstruction, situational understanding, and pose estimation. Using the Manhattan world hypothesis and the Atlanta world hypothesis can improve the robustness and accuracy of visual SLAM.

However, whether it is the Manhattan World Hypothesis or the Atlanta World Hypothesis, there is a limitation that it can be applied only to a normal ideal artificial environment. In practice, in the most common situation, when you include multiple box worlds, each box world all has a different orientation, and the orientations of boxes in different locations can be very adjacent. If the tested box world is unnaturally parallel to a single Manhattan world or Atlanta world, performance and robustness can be much worse.

With respect to the above-mentioned problem, one aspect of the present embodiment is to re-identify the point, line, and structure (corresponding to the Manhattan frame) information among the history map information using a highly efficient and accurate method. Another aspect is to divide the scene according to location into multiple local Manhattan worlds rather than a single Manhattan world or Atlanta world. Each local Manhattan world consists of a set of Manhattan frames (represented in the coordinate system of the underlying local box world in the current image camera coordinate system) that are related to each other by a spatial common view. In other words, each new Manhattan frame is re-identified by the local box world and the corresponding spatial common view correlated Manhattan frame. In addition, if the image information of the camera at some angles is insufficient for the Manhattan frame estimation, the estimation may be inaccurate, and it is not necessary to test the Manhattan frame every time. Testing of the Manhattan frame can be permitted when a sufficient straight line has been found and a constant relationship exists between the relative rotation of the Manhattan frame and the output pose of the SLAM system. In this way, by allowing the algorithm to more effectively use feature information of a dimension different from the history map information, it is possible to more accurately and robustly suppress the accumulation of errors.

Specifically, in this embodiment, a novel visual-inertial SLAM method is presented, and the method is based on the visual relationship SLAM method and introduces a point, line and Manhattan frame re-identification method. To briefly summarize the method of re-identification of points, lines, and Manhattan frames: first, we detect point and line features from the current image frame, and coarse-to-fine the Manhattan frame using the line features to estimate the Manhattan frame of the current image frame. acquiring a frame; Next, constructing a spatiotemporal sensitive global submap using the global map generated by the visual-inertial SLAM; Finally, using a spatiotemporal-sensitive global submap and pose-guided validation and matching method, re-identify points and lines, and sequentially verify temporal coherence and spatiotemporal coherence of the Manhattan frame, i.e., re-identifying the Manhattan frame, with points, lines and adding the re-identification constraint relationship of the Manhattan frame or the like to the cost function of the bundle adjustment to solve the pose of the device and the three-dimensional map.

As shown in Fig. 1, the execution of the SLAM method provided by this embodiment is by the SLAM system shown in the figure, and the system includes an input module, a front-end module, a rear-end module, a re-identification module and an output module. . Among them, the input module includes a relationship measurement device and a binocular camera, which are used by the user to acquire inertial measurement data and image data, respectively. The front-end module includes a preliminary integration unit of inertial measurement data, a point-line test and tracking matching unit, and a Manhattan frame estimation unit. The back-end module includes a global bundle coordination unit and a local bundle coordination unit. The re-identification module includes a spatiotemporal sensitive global sub-map creation unit and a pose guide verification and matching unit. The output module includes a real-time pause output unit. The function of each unit included in each module will be described in detail in the following description of the SLAM method.

FIG. 2 is a flowchart of the SLAM method provided by this embodiment. As shown in FIG. 2, the method includes obtaining a structure of a current image frame (S201), restructuring based on the structure of the current image frame. obtaining an identification constraint (S202), and obtaining a SLAM result based on the structure re-identification constraint (S203). The method can better suppress error accumulation by introducing a structure re-identification constraint in the bundle adjustment process for obtaining SLAM results, and can improve accuracy and robustness in obtaining SLAM results.

In this embodiment, the SLAM method will be described with the same structure as the Manhattan frame. That is, the SLAM method will be described by replacing "structure" with "manhattan frame", and it goes without saying that the protection scope of the present embodiment is not limited thereby.

Furthermore, the method comprises the steps of:

In step (1), a preliminary integration constraint of the inertial measurement data corresponding to the current image frame is obtained, and the first class point corresponding to the current image frame, the dot-line tracking constraint of the first class line and the first point-line matching get restrictions.

In this case, the first class point and the first class line are points and lines included in the image frame before the current image frame, respectively, and the second class line and the second class line are points and lines not included in the previous image frame, respectively. . Based on the frame before the current image frame, it can be seen that the second class point and the second class line are newly increased points and lines, that is, newly increased feature information. In the information processing process of the current image frame, the first class point and the first class line are tracked and matched, and the second class point and the second class line are tested.

Specifically, in the step of acquiring the image data (ie, the current image frame) and the inertial measurement data of the binocular camera, a pre-integration unit of the inertial measurement data is configured to measure the inertia between the current image frame and the previous image frame. Pre-integrate the data. The dot-line detection and tracking matching unit tracks the point and line features of the previous frame image (ie, first class points and first class lines), and in the area where there are no point and line features, new point and line features (ie, After detecting the second class point and the second class line), the first class point and the second class line) are detected, and then all points and line features of the current image frame are detected from the image of another camera based on the limit constraint. Obtain a first point-line matching constraint of a first class line, and obtain a second point-line matching constraint for a second class point and a second class line.

In step (2), a second class point and a second class line of the current image frame are obtained, and a Manhattan frame of the current frame is obtained based on the first class line and the second class line.

Specifically, coarse-to-fine estimation is performed on the Manhattan frame using all line features of the current image frame (ie, the first class line and the second class line), and the Manhattan frame of the current image frame is obtained.

In step (3), a second point-line matching constraint, a dot-line re-identification constraint and a Manhattan re-identification constraint are obtained based on the second class point, the second class line, and the Manhattan frame.

Specifically, in the re-identification module, a space-time-sensitive global sub-map construction unit (or Spatial Temporal Sensitive (STS) sub-global map construction unit) is configured with 3 corresponding to the current image frame. A time- and space-sensitive global submap (time- and space-sensitive global submap) based on the spatial distribution relationship of the dimensional map and the three-dimensional map corresponding to each key image frame at the rear end, and the temporal relationship between the current image frame and each key image frame at the rear end sub-map) is created. The pose guide matching unit acquires the image of the current image frame and the image of the key image frame having a spatial common view through the spatiotemporal-sensitive global sub-map, and compares the dot-line feature of the key image frame with the spatial common view and the Manhattan frame By using, the newly detected dot-line features (ie, second-class points and second-class lines) for the image of the current image frame and the Manhattan frame are re-identified.

In step (4), the SLAM result is based on the preliminary integration constraint, the dot-line tracing constraint, the dot-line re-identification constraint, the first dot-line matching constraint, the second dot-line matching constraint, and the Manhattan frame re-identification constraint. to acquire

Specifically, by performing bundle adjustments based on the preliminary integration constraint, dot-line tracking constraint, dot-line re-identification constraint, and Manhattan frame re-identification constraint in the back-end module, the pose and 3D map are optimized and corresponding SLAM results are obtained. and output the corresponding real-time pose through the output module.

In this embodiment, the current image frame is a key image frame, and a preliminary integration constraint, a dot-line matching constraint, a first dot-line matching constraint, a second dot-line matching constraint, a dot-line re-identify constraint, and a Manhattan frame re-constraint How to obtain a SLAM result based on an identification constraint:

obtaining an initial SLAM result corresponding to the current image frame;

All key image frames before the current image frame, based on the preliminary integral, dot-line tracing constraint, first dot-line matching constraint, second dot-line matching constraint, dot-line re-identification constraint, and Manhattan frame re-identification constraint Combine the corresponding preliminary integration constraint, dot-line tracing constraint, first dot-line match constraint, second dot-line match constraint, dot-line re-identification constraint and Manhattan frame re-identification constraint, and bundle the global bundle into the initial SLAM result. performing an adjustment to obtain a global bundle adjusted SLAM result; and

Pre-integration constraint, dot-line tracking constraint, first dot-line matching constraint and dot-line re-constraint of all image frames within the fourth preset time sequence sliding window corresponding to the current image frame and obtaining a SLAM result by performing a local bundle adjustment on the result of the global bundle adjusted SLAM, based on the identification constraint.

In this case, by determining whether the current image frame is a key image frame, it can be determined whether the point and line information of the newly added current image frame reaches a preset quantity based on the previous key image frame, and the preset quantity is reached. In one case, the current image frame is the key image frame. It will be explained that the determination of the key image frame may be based on other preset rules such as determining, for example, image frames having the same interval in the time sequence as the key image frame.

Here, preliminary integration constraint, dot-line tracking constraint, first dot-line matching constraint, second dot-line matching constraint, dot-line re-identification constraint, and Manhattan frame re-identification constraint corresponding to all key image frames before the current image frame The identification constraint, and the preliminary integration constraint, the dot-line tracking constraint, the first dot-line matching constraint and the dot-line re-identification constraint of all image frames within the fourth preset time sequence sliding window corresponding to the current image frame are the initial SLAM can be obtained from the results. That is, it may be obtained from the global map corresponding to the initial SLAM result.

Specifically, when determining the current image frame as a key image frame, in the bundle adjustment process, global bundle adjustment is first performed on the initial SLAM result to obtain a global bundle adjusted SLAM result, and then the global bundle adjusted SLAM result is added to the global bundle adjusted SLAM result. SLAM results are obtained by performing local bundle adjustments. When it is determined that the current image frame is not a key image frame, the preliminary integration constraint, the dot-line tracking constraint, and the first dot-line matching constraint of all image frames in the fourth preset time sequence sliding window corresponding to the current image frame are determined. and a local bundle adjustment to the initial SLAM result based directly on the dot-line re-identification constraint to obtain the SLAM result.

For example, shown in FIG. 3 is a specific flowchart of a SLAM method provided by this embodiment, which may include the following several steps:

First, image data and inertial measurement data of the binocular camera are acquired, and preliminary integration is performed on the data between the current image frame and the previous image frame collected by the inertial measurement device, and the point and line characteristics of the image of the previous frame are traced. Then, after detecting new point and line features in a region without point and line features, all points and line features of the current image frame are detected from the image of another camera based on the limit constraint, and all line features of the current image frame are detected. is used to perform coarse-to-fine estimation on the Manhattan frame. Optionally, this may be done only for key image frames.

Next, the spatial distribution relationship of the 3D map corresponding to the image of the current image frame and the 3D map corresponding to the image of each key image frame at the rear end, and the image of the current image frame and the image of each key image frame at the rear end Construct a spatiotemporal-sensitive global submap based on the temporal relationship, obtain an image of the current image frame and an image of a key image frame having a spatial common view through the spatiotemporal sensitive global submap, and obtain a key image frame having a spatial common view. Through the dot-line feature of the image and the Manhattan frame, the newly detected dot-line feature and the Manhattan frame for the image of the current image frame are re-identified.

Finally, when a new key image frame is generated, the tracking matching and re-identification constraints of the dot-line features of all key image frames, the relative rotation constraints between the image with a spatial common view and the Manhattan frame, and preliminary integration of the inertial measurement data Bundling constraints to optimize poses and 3D maps with global scope. In addition, by performing bundle adjustment for tracking matching and re-identification constraints of dot-line features of all image frames within the time sequence sliding window and pre-integration constraints of inertial measurement data, it optimizes poses and 3D maps in the local range of the time domain. do.

In this embodiment, the method of obtaining the Manhattan frame of the current image frame based on the first class line and the second class line is:

finding a first vanishing direction of the Manhattan frame through a two-line segment minimum solution set algorithm based on any two of the first class line and the second class line;

A second vanishing direction of the Manhattan frame is obtained by performing conformal sampling on a vertical circle of the first vanishing direction, and a third vanishing direction of the Manhattan frame is obtained based on the first vanishing direction and the second vanishing direction. obtaining a vanishing direction, wherein the diameter of the vertical column and the diameter of the equivalent spherical surface of the current image frame are the same;

obtaining an initial Manhattan frame based on the first vanishing direction, the second vanishing direction, and the third vanishing direction;

Each of the inner line segments of the three vanishing points of the Manhattan frame in the current image frame is acquired, and the three vanishing directions of the initial Manhattan frame are optimized based on the inner line segments of the three vanishing points to optimize the Manhattan frame. acquiring a frame.

Specifically, a structured environment (generally an artificial scene) exhibits certain regularities, such as parallelism and orthogonality, for example. As shown in Fig. 4, the Manhattan world refers to a scene with three mutually orthogonal vanishing directions on an equivalent sphere, which correspond to three vanishing points on the image plane. In Manhattan World, given all the line segments detected from the calibration image, the goal is to collect them individually as three vanishing points. The Manhattan frame is used extensively in the structural model of the Manhattan World, in which the three axes of the Manhattan frame correspond to three mutually orthogonal vanishing directions of the Manhattan World, and the three-dimensional lines of the Manhattan World are parallel to the three coordinate axes of the entire Manhattan frame. do. Based on this constraint, the three vanishing directions orthogonal to each other can again be described as a rotation between the Manhattan world and the camera frame, that is, the Manhattan frame.

In this embodiment, an estimation process of three vanishing directions (that is, the Manhattan frame) orthogonal to each other will be described in detail. Three vanishing points are estimated based on the detected image segment features, and three vanishing directions orthogonal to each other are calculated by the three vanishing points. Note that the vanishing point can be calculated based on the line clustering result, and when the vanishing point is estimated, the line clusters can be sequentially obtained. Based on this fact, a coarse-to-fine Manhattan frame estimation method is designed, and the method is mainly composed of a coarse estimation module and a fine estimation module. The coarse estimation module performs a full high-speed search for three disappearance directions orthogonal to each other, and provides initial values to the fine estimation module. The fine estimation module further optimizes the results of three vanishing directions orthogonal to each other based on the output of the coarse estimation module. Below, the detailed execution methods of the coarse estimation module and the fine estimation module will be described in detail.

)을 생성한다. 다음으로, 제1 소실 방향의 수직원 상에서 등각 샘플링을 실시하여 맨하탄 프레임의 제2 소실 방향(

)을 획득한다. 마지막으로,

과

의 교차 곱셈에 의하여 제3 소실 방향(

)을 획득한다. 이상의 세 단계를 통하여 다양한 소실 방향의 조합을 획득한 다음, 가설 검증을 통하여 초기 맨하탄 프레임을 결정한다. 즉, 극좌표 격자의 구성을 통하여 가장 바람직한, 서로 직교하는 세 개의 소실 방향이 결정된다. 이로써, 맨하탄 프레임에 대한 coarse 추정이 완료된다.Coarse estimation module: This embodiment uses a two-line minimum solution set to form a hypothesis of the first vanishing direction, sample the second vanishing direction, and calculate the first and second vanishing directions. to create a third vanishing direction. Specifically, first, two mutually orthogonal line segments in the current image frame are arbitrarily selected, and based on the two mutually orthogonal line segments, the first vanishing direction (

) is created. Next, conformal sampling is performed on the vertical line of the first vanishing direction to form the second vanishing direction (

) is obtained. Finally,

class

The third vanishing direction (

) is obtained. After obtaining a combination of various vanishing directions through the above three steps, the initial Manhattan frame is determined through hypothesis testing. That is, through the configuration of the polar coordinate lattice, the most desirable three vanishing directions orthogonal to each other are determined. Accordingly, coarse estimation for the Manhattan frame is completed.

,

,

)에 의하여 세 개의 소실 방향에 대한 coarse 추정치를 계산하고, 이하의 공식을 통하여 세 개의 소실 방향(

,

,

)을 최적화한다:Fine estimation module: As shown in FIG. 5 , first, three vanishing points in the current image frame of the initial Manhattan frame are obtained, and inner line segments corresponding to the three vanishing points are respectively obtained. Specifically, when the included angle between the line connecting the midpoint of any one line segment and the vanishing point and the line segment is less than or equal to a preset angle, the line segment is determined as the inner line segment of the vanishing point. Of course, there may be multiple internal line segments of one vanishing point, and the size of the preset angle may be set according to actual needs. Next, the inner segments of the three vanishing points of the initial Manhattan frame (corresponding to relatively thick segments in the plane) (

,

,

) calculates the coarse estimates for the three vanishing directions, and the three vanishing directions (

,

,

) to optimize:

Here, π() is coordinate transformation, that is, transformation from the camera coordinate system to the image coordinate system.

)을 계산할 수 있으며, 이 때

으로서, 마찬가지로 맨하탄 프레임이다.Finally, the rotation between the Manhattan world and the camera frame (i.e. the image frame) by the three vanishing directions being optimized (

) can be calculated, where

As, likewise, the Manhattan frame.

In this embodiment, the method of performing re-identification based on the second class point, the second class line and the Manhattan frame and obtaining the corresponding point-line re-identification constraint and the Manhattan frame re-identification constraint is:

obtaining a key image frame before the current image frame;

obtaining a spatiotemporal sensitive global submap corresponding thereto based on the current image frame and the key image frame;

and obtaining corresponding point-line re-identification constraints and Manhattan frame re-identification constraints by performing re-identification on second class points, second class lines, and Manhattan frames based on the spatiotemporal sensitive global sub-map.

Below, two aspects of the acquisition of spatiotemporal-sensitive global submaps and the implementation of re-identification for the Manhattan frame are described in detail.

In this embodiment, on the basis of the current image frame and the key image frame, the method for obtaining the corresponding spatiotemporal sensitive global submap is:

Based on the time sequence relationship between the current image frame and the key image frame and the spatial relationship between the three-dimensional map corresponding to the current image frame and the three-dimensional map corresponding to the key image frame, to obtain a corresponding, spatiotemporal-sensitive global sub-map includes steps.

Specifically, based on the time sequence relationship between the current image frame and the key image frame and the spatial relationship between the three-dimensional map corresponding to the current image frame and the three-dimensional map corresponding to the key image frame, a corresponding, space-time-sensitive global sub-map is generated. The steps to obtain are:

obtaining a reference key image frame of the current image frame based on a time sequence relationship between the current image frame and the key image frame;

obtaining a two-dimensional spatial distribution between an existing image frame and a key image frame based on a spatial relationship between a three-dimensional map corresponding to the current image frame and a three-dimensional map corresponding to the key image frame;

In a two-dimensional spatial distribution, a key image frame with a reference key image frame and an overlapping region is used as a key image frame with a spatial common view with the current image frame, and a spatiotemporal sensitive global submap based on the key image frame with a spatial common view to acquire

)으로 간주되며, 시간적으로 현재 이미지 프레임과 비교적 멀리 떨어진 공간적 공통 뷰 키 프레임 이미지는 현재 이미지 프레임의 시공간 민감 글로벌 서브 맵, 즉 (

)를 구성한다.Specifically, as shown in Fig. 6, the camera moves clockwise in the left plan view, and the two-dimensional spatial distribution of each key image frame is visualized by a rectangle indicated by a dotted line, which is a three-dimensional space indicated by a cube in the figure on the right. corresponding to the distribution. A key image frame having a spatial distribution and overlapping area (shaded area) of the reference key image frame of the current image frame is a key image frame having a spatial common view (

), and a spatially common view keyframe image that is relatively far away from the current image frame in time

) constitutes

In this embodiment, the method of re-identifying the second class point, the second class line and the Manhattan frame based on the spatiotemporal sensitive global submap is:

obtaining a key image frame having a spatial common view with the current image frame based on the spatiotemporal-sensitive global sub-map;

Obtain a point, a line and a Manhattan frame of a key image frame having a spatial common view, and a second class point, a second class based on the point, line, and Manhattan frame associated with the point, line and spatial common view of the key image frame having a spatial common view and re-identifying the line and the Manhattan frame.

Specifically, re-identifying the Manhattan frame based on the Manhattan frame associated with the spatially common view of the key image frame having the spatially common view comprises:

performing a temporal consistency test on the Manhattan frame based on a relative rotation error between the Manhattan frame of the current image frame and the time-correlated Manhattan frame of the image frame in the corresponding first preset time sequence sliding window;

If the temporal consistency test for the Manhattan frame is passed, the temporal correlation Manhattan frame of the image frame within the sliding window corresponding to the second preset time sequence sliding window corresponding to the current image frame, and the third preset corresponding to the key image frame having a spatially common view obtaining a spatially correlated Manhattan frame of an image frame within a time sequence sliding window, and obtaining a relative rotational drift error between the temporally correlated Manhattan frame and a corresponding key image frame having a spatial common view;

and performing a temporal-spatial consistency test on the Manhattan frame based on the error of the relative rotational drift.

Specifically, in order to cope with a more complex environment and secure robustness in the Manhattan frame estimation, in this embodiment, the Manhattan frame was re-identified using the pose guide verification and matching method of the Manhattan frame as shown in FIG. . Input is the current Manhattan frame and the entire map generated by the visual-inertial SLAM. First, temporal consistency between the Manhattan frame of the current image frame and the corresponding time-related Manhattan frame is tested, and whether the current Manhattan frame is sufficiently robust is checked. Thereafter, the common spatial view correlation Manhattan frame of the Manhattan frame of the current image frame is searched for through the spatiotemporal-sensitive global submap, and it is matched with the Manhattan frame of the current image frame. After the temporal consistency test and matching of the Manhattan frame were implemented, in order to further improve the robustness of the Manhattan frame estimation, the temporal consistency of the Manhattan frame was tested. Finally, the relative rotation set of the Manhattan frame of the current image frame and the corresponding spatial common view correlation Manhattan frame is output to the rear end, and the Manhattan frame constraint is input to the rear end. Below, the temporal consistency test of the Manhattan frame, the matching of the Manhattan frame, and the temporal-spatial consistency test of the Manhattan frame will be described in detail.

In this embodiment, based on the relative rotation error between the Manhattan frame of the current image frame and the time-correlated Manhattan frame of the image frame within the corresponding first preset time sequence sliding window, a method for performing a temporal consistency test on the Manhattan frame silver:

obtaining a corresponding average relative rotation error based on all relative rotation errors corresponding to the first preset time sequence sliding window;

passing the Manhattan frame in a temporal consistency test if the average relative error is less than or equal to the first preset threshold.

)과 거의 유사하다. 이러한 특성을 이용하여, 본 실시예에서는 맨하탄 프레임의 안정성과 강건성에 대한 테스트를 실시하였다. 도 8에 도시된 바와 같이, 타임 시퀀스 슬라이딩 윈도우(즉 제1 타임 시퀀스 슬라이딩 윈도우)에는 연속 이미지 프레임, 즉

이 포함된다. 본 실시예에서는 각각 VIO와 맨하탄 프레임 방법을 이용하여 이미지 프레임(

)과 타임 시퀀스 슬라이딩 윈도우 내의 다른 이미지 프레임 사이의

과

의 상대 회전을 계산한다. 타임 시퀀스 슬라이딩 윈도우 내의

와

사이의 오차(즉, 상대 회전 오차)를 계산하여 맨하탄 프레임의 강건성을 검증하였다.Specifically, in the visual-inertial odometry (VIO) or visual-inertial SLAM method, the relative rotation between image frames within the same time sliding window has a very small error, and the actual value of the relative rotation (

) is almost identical to Using these characteristics, in this embodiment, the stability and robustness of the Manhattan frame were tested. As shown in Fig. 8, the time sequence sliding window (ie, the first time sequence sliding window) has a continuous image frame, that is,

This is included. In this embodiment, using the VIO and Manhattan frame methods, respectively, the image frame (

) and the time sequence between other image frames within the sliding window.

class

Calculate the relative rotation of within a time sequence sliding window

Wow

The robustness of the Manhattan frame was verified by calculating the error between them (ie, relative rotation error).

Specifically, the calculation formula of the average relative rotation error is as follows:

는 회전 벡터 오차가 각도 스칼라 오차로 변환되는 함수이며, 구체적인 공식은 하기와 같다:At this time,

is a function in which the rotation vector error is converted to an angle scalar error, and the specific formula is as follows:

는 현재 윈도우 프레임과 i번째 이미지 프레임의 맨하탄 프레임 방법으로 획득한 상대 회전을 나타내며,

는 현재 이미지 프레임과 i번째 이미지 프레임의 VIO 방법으로 획득한 상대 회전을 나타낸다.

가 제1 사전 설정 임계값 이하인 경우, 즉 맨하탄 프레임이 시간 일관성 테스트를 통과하면, 맨하탄 프레임과 이에 대응되는 공간적 공통 뷰 상관 맨하탄 프레임을 매칭한다.

represents the relative rotation obtained by the Manhattan frame method of the current window frame and the i-th image frame,

denotes the relative rotation of the current image frame and the i-th image frame obtained by the VIO method.

If is less than or equal to the first preset threshold, that is, if the Manhattan frame passes the temporal consistency test, the Manhattan frame and the corresponding spatial common view correlation Manhattan frame are matched.

In this embodiment, the method of performing a time-space consistency test for the Manhattan frame based on the relative rotation drift error is:

obtaining a ratio of relative rotational drift errors corresponding to the second preset time sequence sliding window and the third preset time sequence sliding window corresponding to a relative rotational drift error equal to or greater than a second preset threshold value;

if the ratio is less than or equal to a third preset threshold, causing the Manhattan frame to pass a spatiotemporal consistency test.

)를 생성하고, (

)는 상대 회전의 실제값(

)과 근사하다. 이러한 특성을 이용하여, 맨하탄 프레임의 안정성과 강건성을 추가적으로 테스트한다. 제2 사전 설정 타임 시퀀스 슬라이딩 윈도우는 (

) 및 이와 시간적으로 인접한 다수 개의 이미지 프레임을 포함하며, 제3 사전 설정 타임 시퀀스 슬라이딩 윈도우는 (

) 및 이와 시간적으로 인접한 다수 개의 이미지 프레임을 포함한다.Specifically, when the Manhattan frame satisfies temporal coherence and matching, in order to further improve the robustness of the Manhattan frame estimation, temporal-spatial coherence is checked. As shown in Fig. 9, the VIO method uses a consistent drift (

), and (

) is the actual value of the relative rotation (

) is close to Using these characteristics, the stability and robustness of the Manhattan frame are further tested. The second preset time sequence sliding window is (

) and a plurality of image frames temporally adjacent thereto, and the third preset time sequence sliding window is (

) and a plurality of image frames temporally adjacent thereto.

과

를 이용하여 상대 회전 드리프트(

)를 계산하며, 공식은 하기와 같다:Specifically, in two different time-sequence sliding windows, the Manhattan frame and matched

class

Relative rotation drift (

), the formula is as follows:

) 이하의 다른 매칭된 맨하탄 프레임 사이의 상대 회전 드리프트 오차의 수를 계산한다.Next, the current image and a second preset threshold (

) below, calculate the number of relative rotation drift errors between different matched Manhattan frames.

는 두 개의 타임 시퀀스 슬라이딩 윈도우 사이의, 맨하탄 프레임 매칭된 그룹을 나타낸다.

denotes a Manhattan frame matched group between two time sequence sliding windows.

Finally, in this example, the ratio is calculated by the following formula:

은 두 개의 상이한 슬라이딩 윈도우 간의, 맨하탄 프레임 매칭의 총 수를 나타낸다.

이 제3 사전 설정 임계값 이하이면, 맨하탄 프레임이 시간-공간 일관성을 만족하는 것으로 판단한다.

represents the total number of Manhattan frame matches between two different sliding windows.

If it is less than or equal to this third preset threshold, it is determined that the Manhattan frame satisfies the temporal-spatial coherence.

Furthermore, similar to most sight-inertial SLAMs, the goal of this embodiment is to estimate the camera's state, which is not revealed, including camera pose, velocity, and a three-dimensional map.

로 설명된다고 가정한다. 각 3차원 맵의 점(

), 선(

)에 대하여, 추적 및 재식별 방법에 의하여 다수의 이미지 프레임 내에서 관찰을 수행한다. 관성 측정은 상대 운동 제약의 제공에 있어서도 매우 중요하므로, 관성 측정에 대한 예비 적분에 의하여 관성 측정 유닛의 상태(

)를 추정하며, 이 때

는 각각 속도와 바이어스를 나타낸다.camera pose

It is assumed to be described as Points on each 3D map (

), line(

), observations are performed within multiple image frames by tracking and re-identification methods. Since the inertia measurement is also very important in providing the relative motion constraint, the state of the inertial measurement unit (

) is estimated, at this time

are the speed and bias, respectively.

) 간의 관성 측정(

), 점-선 측정(

)과 재식별(

)의 시각적 측정(

)을 처리한다. 슬라이딩 윈도우 내에서 가장 오래된 프레임이 슬라이딩 윈도우 외부로 반출될 때, 이와 대응되는 시각적 및 관성 측정은 번들 조정된 선험적 오차(

)로 변환된다. 로컬 번들 조정된 함수는 하기와 같이 정의된다:Image frames within sliding windows based on local bundle coordinated non-linear optimization framework (

) to measure the inertia between

), point-line measurement (

) and re-identification (

) a visual measure of (

) is processed. When the oldest frame within a sliding window is taken out of the sliding window, the corresponding visual and inertial measurements are subject to bundle-adjusted a priori error (

) is converted to A local bundle coordinated function is defined as:

) 간의 관성 측정(

), 점-선 추적(

) 및 점-선 재식별(

), 시각적 측정(

), 맨하탄 프레임 재식별 상대 회전(

)을 처리한다. 타임 시퀀스 슬라이딩 윈도우 내의 가장 오래된 키 이미지 프레임이 슬라이딩 윈도우로부터 반출될 때, 이에 대응되는 선험 프레임은 전역 번들 조정에서 상대 포즈(

)로 변환된다. 전역 번들 조정의 함수는 하기와 같이 정의된다:All key image frames (

) to measure the inertia between

), dot-line trace (

) and dot-line re-identification (

), a visual measurement (

), Manhattan frame re-identification relative rotation (

) is processed. When the oldest key image frame in the time sequence sliding window is exported from the sliding window, the corresponding a priori frame is set to the relative pose (

) is converted to The function of global bundle coordination is defined as follows:

10 shows the trajectory motion of the camera in the case of including four partial Manhattan worlds. 437th, 1081th, 1446th, and 1709th of the left figure correspond to the image on the right, respectively. 437th and 1081th correspond to the same local Manhattan world (4), and the right figure also shows a spatial common view of 437th and 1081th, so the calculated relative rotation angle error after Manhattan frame re-identification is that estimated by visual-inertia SLAM will be smaller than On the contrary, since 1446th and 1709th correspond to different local Manhattan worlds, and the right figure also shows that there is no spatial common view at 1446th and 1709th, the relative rotation angle error calculated after Manhattan frame matching is calculated by visual-inertia SLAM will be larger than estimated.

11 is a structural block diagram of a SLAM device provided by the present embodiment. As shown in FIG. 11 , the device 1000 includes a structure acquisition module 1001, a structure re-identification constraint acquisition module 1002, and a SLAM. A result acquisition module 1003 comprising:

The structure obtaining module 1001 is used to obtain the structure of the current image frame;

The structure re-identification constraint obtaining module 1002 is used to obtain the structure re-identification constraint based on the structure of the current image frame;

The SLAM result obtaining module 1003 is used to obtain a SLAM result based on the structure re-identification constraint.

In the method provided by this embodiment, it is possible to better suppress the accumulation of errors by introducing the structure re-identification constraint in the bundle adjustment process for obtaining the SLAM result, and to improve the accuracy and robustness of the SLAM result.

In this embodiment, the device comprises:

obtain points and lines of the current image frame;

a dot-line re-identification constraint module for re-identifying points and lines of the current image frame to obtain a point-line re-identification constraint;

The SLAM Result Acquisition module specifically

It is used to obtain SLAM results based on the dot-line re-identification constraint and the structural re-identification constraint.

In this embodiment, the structure acquisition module includes:

obtain an initial structure based on the line of the current image frame;

It is used to optimize the initial structure and obtain the structure based on a line that satisfies a preset condition among the current image frame.

In this embodiment, the structure re-identification constraint obtaining module is specifically

obtain a global submap based on the current image frame and the key image frame;

It is used to re-identify the structure based on the global sub-map, and to obtain the structure re-identification constraint.

In this embodiment, the structure re-identification constraint obtaining module is also

It is used to obtain a global sub-map based on a time sequence relationship between the current image frame and the key image frame, and a spatial relationship between a three-dimensional map corresponding to the current image frame and a three-dimensional map corresponding to the key image frame.

In this embodiment, the structure re-identification constraint obtaining module is also

obtaining a reference key image frame of the current image frame based on a time sequence relationship between the current image frame and the key image frame;

Based on the spatial relationship between the current image frame and the corresponding three-dimensional map and the key image frame and the corresponding three-dimensional map, the key image frame having the overlapping area with the reference key image frame in the spatial distribution is divided into the current image frame and the spatial common view. setting the branch as a key image frame; and

and obtaining a global sub-map based on the key image frame having a spatial common view.

In this embodiment, the structure re-identification constraint obtaining module is also

obtaining a key image frame having a common spatial view with the current image frame based on the global sub-map;

and re-identifying the structure based on the common view correlation structure of the key image frame having a common spatial view.

In this embodiment, the structure re-identification constraint obtaining module is also

based on the structure of the current image frame and the temporal correlation structure of the image frame in the first preset time sequence sliding window corresponding thereto, perform a temporal consistency test on the structure;

If the temporal consistency test for the structure is passed, the temporal correlation structure of the image frame in the sliding window corresponding to the second preset time sequence corresponding to the current image frame, and the third preset time sequence corresponding to the key image frame having a spatially common view match the structure based on the spatial common view correlation structure of the image frame within the sliding window;

After matching is implemented, a temporal consistency test is performed on the structure based on a key image frame having a temporal correlation structure and a corresponding spatial common view.

In this embodiment, the structure re-identification constraint obtaining module is also

obtain a relative rotation error between the structure of the current image frame and the temporal correlation structure of the image frame within the corresponding first preset time sequence sliding window;

The first preset time sequence is used to perform a temporal consistency test on the structure based on all relative rotation errors corresponding to the sliding window.

In this embodiment, the structure re-identification constraint obtaining module is also

obtain a relative rotational drift error between a key image frame having a temporal correlation structure and a corresponding spatially common view;

obtain a relative rotation drift error corresponding to the second preset time sequence sliding window and the third preset time sequence sliding window;

Based on the relative rotation drift error corresponding to the second preset time sequence sliding window and the third preset time sequence sliding window, it is used to perform a time-spatial consistency test for the structure.

In this embodiment, when the current image frame is a key image frame, the SLAM result obtaining module specifically

obtain a global bundle adjusted SLAM result based on the dot-line re-identification constraint and the structure re-identification constraint;

Based on the dot-line re-identification constraint of all image frames in the fourth preset time sequence sliding window corresponding to the current image frame, local bundle adjustment is performed on the global bundle adjusted SLAM result to obtain a SLAM result. do.

Hereinafter, reference is made to FIG. 12 in which a structural diagram of an electronic device (eg, a terminal equipment or a server executing the method shown in FIG. 2 ) 1100 for implementing the present embodiment is shown.

The electronic device of the present embodiment includes a mobile phone, a notebook computer, a digital broadcast transmitter, a personal digital assistant (PDA), a tablet (PAD), a portable multimedia player (PMP), a GPS (eg, a vehicle navigation terminal), a wearable device, and the like. It may include, but is not limited to, a mobile terminal of a mobile terminal and a fixed terminal such as a digital TV and a desktop PC. The electronic device shown in FIG. 12 is only an exemplary embodiment, and does not limit the function and use range of the present exemplary embodiment.

The electronic device includes a memory and a processor, wherein the memory stores a procedure of the method described in the embodiment of each method; The processor is configured to execute the procedure stored in the memory. Here, the processor may be referred to as a processing unit 1101 to be described below, and the memory may include a read-only memory (ROM) 1102 to be described below, and a random access memory (RAM) 1103 and a memory device 1108 , which will be described below. ) may include at least one of, specifically as follows:

As shown in FIG. 12 , the electronic device 1100 can perform various appropriate operations and processing according to a procedure stored in the ROM 1102 or a procedure loaded into the RAM 1103 from the memory device 1108 . processing unit (eg, CPU, GPU, etc.) 1101 . The RAM 1103 further includes various procedures and data necessary for the operation of the electronic device 1100 . The processing unit 1101 , the ROM 1102 , and the RAM 1103 are connected to each other via a bus 1104 . An input/output (I/O) interface 1105 is also coupled to the bus 1104 .

input devices 1106, including generally a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, and the like; an output device 1107 including a liquid crystal display (LCD), a speaker, a vibrator, and the like; memory device 1108, including magnetic tape, hard disk, and the like; and a communication device 1109 may be coupled to the I/O interface 1105 . The communication device 1109 may allow the electronic device 1100 to communicate with another device by wire or wirelessly to exchange data. Although FIG. 12 illustrates an electronic device including various devices, it will be understood that not all illustrated devices may be implemented or provided. More or fewer devices may generally be implemented or equipped.

In particular, according to this embodiment, the process described with reference to the flowchart may be implemented as a computer software program. For example, the present embodiment includes a computer program product including a computer program stored in a non-transitory computer readable medium, wherein the computer program includes program code for executing the method shown in the flowchart. In this embodiment, the computer program may be downloaded and installed from the network via the communication device 1109 , installed from the memory device 1108 , or installed from the ROM 1102 . When the computer program is executed by the processing device 1101, it executes the functions defined in the method of this embodiment.

For reference, the computer-readable storage medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination of both. A computer-readable storage medium may include, but is not limited to, for example, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or element, or any combination thereof. More specific examples of computer-readable storage media include an electrical connection comprising one or more conductors, a portable computer magnetic disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable and programmable ROM (EPROM). or flash memory), optical fiber, portable compact disk ROM (CD-ROM), optical memory device, magnetic memory device, or any suitable combination of the foregoing examples. A computer-readable storage medium may be a tangible medium containing or storing a program, and the program may be used by or in combination with an instruction execution system, apparatus, or element. Further, the computer readable signal may be a data signal included in baseband or transmitted as part of a carrier wave, and carries therein a computer readable program code. The transmitted data signal may use various forms, and may include, but is not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the foregoing forms. A computer-readable signal medium is any computer-readable medium other than a computer-readable storage medium, wherein the computer-readable signal medium is used by or in combination with a system, apparatus, or element that executes instructions. You may send, deliver, and transmit programs. A program card included in a computer readable medium may be transmitted using any suitable medium, which may include, but is not limited to, wire, optical cable, RF (frequency), etc., or any suitable combination of the foregoing. doesn't happen

In some embodiments, the user terminal and the server may perform communication using any known or to-be researched and developed network protocol, such as HTTP (HyperText Transfer Protocol, Hypertext Transfer Protocol), and may be digitally in any form or medium. It may be interconnected with data communications (eg, a communications network). Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), networks of networks (eg, the Internet) and end-to-end networks (eg, ad hoc end-to-end networks), and any known public domain. Alternatively, it may include a network to be researched and developed in the future.

The aforementioned computer readable medium may be included in the aforementioned electronic device; It may exist alone and may not be installed in the electronic device.

One or more programs are loaded in the aforementioned computer readable medium, and when the aforementioned one or more programs are executed by the electronic device, the electronic device includes:

obtain the structure of the current image frame; obtain a structure re-identification constraint based on the structure of the current image frame; Based on the structural re-identification constraint, SLAM results are obtained.

One or more programming design languages or a combination thereof may generate computer program code for implementing the operation of the present embodiment, wherein the programming design language may include an object-oriented programming design language such as Java, Smalltalk, or C++. However, the present invention is not limited thereto, and may include a procedural programming design language such as "C" language or a similar programming design language. The program code may run entirely on the user's computer, partially on the user's computer, as a separate software package, some on the user's computer and some on a remote computer, or completely on a remote computer or server. may be When running on a remote computer, the remote computer may be connected to the user's computer through any kind of network, such as a local area network (LAN) or wide area network (WAN), or an external computer (eg, to an Internet service provider). may be connected to the Internet).

The flowcharts and block diagrams in the drawings illustrate possible structures, functions, or operations of systems, methods, and computer programming products according to respective embodiments. In this regard, each block in the flowchart or block diagram may represent one module, program piece, or portion of code, wherein the module, program piece, or portion of code is one or more for implementing a specified logic function. contains executable instructions. In some alternative implementations, the functions indicated in the blocks may occur in a different order than the functions indicated in the figures. For example, two blocks indicated as being connected may actually be basically executed simultaneously, and in some cases, they may be executed in the opposite order, depending on the function involved. It should be noted that each block in the block diagram and/or flowchart, and a combination of blocks in the block diagram and/or flowchart may be implemented using a software-based dedicated system that performs specified functions or operations, or dedicated software And it can be implemented using a combination of computer instructions.

Modules or units described in this embodiment may be implemented in a software manner or may be implemented in a hardware manner. Here, the name of the module or unit may not constitute a limitation on the unit itself depending on circumstances. For example, the first constraint obtaining module may be described as “a module obtaining the first constraint”.

The functions described above may be implemented, at least in part, by one or more hardware logic components. By way of example, and not limitation, preferred types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and application-specific integrated circuits (ASICs). -specific standard product (ASSP), single-chip system (system on chip, SOC), complex programmable logic device (complex programmable logic device, CPLD), and the like.

In the context of the present embodiment, a machine-readable medium is a tangible medium that can contain or store a program provided on or used in combination with an instruction execution system, apparatus, or device. can The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, ultraviolet, or semiconductor systems, apparatus or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include one or more wire electrical connections, portable computer disks, hard disks, RAM, removable ROM (EPROM or high-speed flash memory), optical fiber, compact disk ROM (CD-ROM). , an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

Those skilled in the art can clearly understand that, for convenience and conciseness of description, they can refer to corresponding steps described in the above-described method embodiments for specific methods implemented when the computer-readable medium described above is executed by an electronic device, so that the repetition A description will be omitted.

The above-described embodiments are merely examples, and do not limit the technical scope in any way. For brevity of the specification, descriptions of well-known electronic components, control systems, software, and other functional aspects have been omitted. In addition, the connections or connecting members of the lines between the components shown in the drawings exemplify functional connections and/or physical or circuit connections, and in an actual device, various functional connections, physical connections that are replaceable or additional It may be implemented as a connection, or circuit connections.

Meanwhile, the above-described embodiments can be written as a program that can be executed on a computer, and can be implemented in a general-purpose digital computer that operates the program using a computer-readable recording medium. In addition, the structure of data used in the above-described embodiments may be recorded in a computer-readable recording medium through various means. The computer-readable recording medium includes a storage medium such as a magnetic storage medium (eg, a ROM, a floppy disk, a hard disk, etc.) and an optically readable medium (eg, a CD-ROM, a DVD, etc.).

Those of ordinary skill in the art related to the present embodiment will understand that the embodiment may be implemented in a modified form without departing from the essential characteristics of the above description. Therefore, the disclosed embodiments are to be considered in an illustrative rather than a restrictive sense. The scope of the rights is indicated in the claims rather than the above description, and all differences within the scope equivalent thereto should be construed as included in the present embodiment.

Claims (14)

In the simultaneous localization and mapping (SLAM) method,
obtaining a structure of a current image frame;
obtaining a structure re-identification constraint based on the structure of the current image frame; and
and obtaining a SLAM result based on the structural re-identification constraint. The method of claim 1,
obtaining points and lines of the current image frame;
Re-identifying the points and lines of the current image frame to obtain a point-line re-identification constraint,
Acquiring a SLAM result based on the structure re-identification constraint comprises:
and obtaining a SLAM result based on the dot-line re-identification constraint and the structural re-identification constraint. The method of claim 1,
The step of obtaining the structure of the current image frame includes:
obtaining an initial structure based on a line of the current image frame; and
and obtaining the structure by optimizing the initial structure based on a line that satisfies a preset condition among the current image frame. The method of claim 1,
The step of obtaining a structure re-identification constraint based on the structure of the current image frame includes:
obtaining a global sub-map based on the current image frame and the key image frame;
and re-identifying the structure based on the global sub-map and obtaining the structure re-identification constraint. 5. The method of claim 4,
Acquiring a global submap based on the current image frame and the key image frame includes:
A global submap is generated based on a time sequence relationship between the current image frame and the key image frame, and a spatial relationship between a three-dimensional map corresponding to the current image frame and a three-dimensional map corresponding to the key image frame. Simultaneous localization and mapping method comprising the step of acquiring. 6. The method of claim 5,
Acquiring a global submap based on a time sequence relationship between the current image frame and the key frame and a spatial relationship between a 3D map corresponding to the current image frame and a 3D map corresponding to the key image frame includes:
obtaining a reference key image frame of the current image frame based on a time sequence relationship between the current image frame and the key image frame;
Based on the spatial relationship between the three-dimensional map corresponding to the current image frame and the three-dimensional map corresponding to the key image frame, a key image frame having an overlapping area with the reference key image frame in the spatial distribution is divided into the current image frame and the current image frame. setting a key image frame having a spatially common view; and
and obtaining the global sub-map based on a key image frame having the spatially common view. 5. The method of claim 4,
Re-identifying the structure based on the global sub-map comprises:
obtaining, based on the global sub-map, a key image frame having a spatially common view of the current image frame; and
based on the spatially common view correlation structure of key image frames having the spatially common view, re-identifying the structure. 8. The method of claim 7,
Re-identifying the structure based on the spatial common view correlation structure of the key image frame having the spatial common view comprises:
performing a temporal consistency test on the structure of the current image frame and the temporal correlation structure of a sliding window corresponding to a first preset time sequence;
If the temporal consistency test for the structure is passed, the temporal correlation structure of the image frame in the sliding window of the second preset time sequence corresponding to the current image frame, and the third preset corresponding to the key image frame having the spatial common view matching the structure based on the spatial common view correlation structure of the image frame within the time sequence sliding window; and
after implementing matching, performing a temporal-spatial consistency test on the structure based on the temporal correlation structure and a key image frame having a corresponding spatial common view. 9. The method of claim 8,
The step of performing a temporal consistency test on the structure of the current image frame based on the structure of the current image frame and the temporal correlation structure of the image frame in the first preset time sequence sliding window corresponding thereto may include:
obtaining a relative rotation error between the structure of the current image frame and the temporal correlation structure of the image frame in the corresponding first preset time sequence sliding window; and
and performing a temporal consistency test on the structure based on all relative rotation errors corresponding to the first preset time sequence sliding window. 9. The method of claim 8,
The step of performing a temporal-spatial consistency test for the structure based on the temporal correlation structure and a key image frame having a corresponding spatial common view includes:
obtaining a relative rotation drift error between the temporal correlation structure and a key image frame having a corresponding spatial common view;
obtaining a relative rotation drift error corresponding to a second preset time sequence sliding window and the third preset time sequence sliding window; and
performing a temporal-spatial consistency test on the structure based on a relative rotation drift error corresponding to the second preset time sequence sliding window and the third preset time sequence sliding window; Estimation and mapping methods. 3. The method of claim 2,
When the current image frame is a key image frame, obtaining a SLAM result based on the dot-line re-identification constraint and the structure re-identification includes:
obtaining a global bundle adjusted SLAM result based on the dot-line re-identification constraint and the structure re-identification constraint; and
Based on the dot-line re-identification constraint of all image frames in the fourth preset time sequence sliding window corresponding to the current image frame, performing local bundle adjustment on the global bundle adjusted SLAM result to obtain a SLAM result; Including, simultaneous localization and mapping method. In a simultaneous localization and mapping (SLAM) device,
a structure obtaining module for obtaining a structure of the current image frame;
a structure re-identification constraint obtaining module for obtaining structure re-identification constraints based on the structure of the current image frame; and
Simultaneous localization and mapping (SLAM) device comprising a; SLAM result acquisition module for acquiring a SLAM result based on the structure re-identification constraint. Memory; and
including a processor;
a computer program is stored in the memory;
The electronic device, wherein the processor is used to implement the method according to any one of claims 1 to 11 by executing the computer program. A computer readable storage medium having stored thereon a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images