CN113763470A - RGBD visual inertia simultaneous positioning and map construction with dotted line feature fusion - Google Patents

RGBD visual inertia simultaneous positioning and map construction with dotted line feature fusion Download PDF

Info

Publication number
CN113763470A
CN113763470A CN202110914560.8A CN202110914560A CN113763470A CN 113763470 A CN113763470 A CN 113763470A CN 202110914560 A CN202110914560 A CN 202110914560A CN 113763470 A CN113763470 A CN 113763470A
Authority
CN
China
Prior art keywords
line
visual
rgbd
dotted line
inertia
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110914560.8A
Other languages
Chinese (zh)
Inventor
赵良玉
朱叶青
金瑞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Institute of Technology BIT
Original Assignee
Beijing Institute of Technology BIT
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Institute of Technology BIT filed Critical Beijing Institute of Technology BIT
Priority to CN202110914560.8A priority Critical patent/CN113763470A/en
Publication of CN113763470A publication Critical patent/CN113763470A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/05—Geographic models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G06T7/248—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/269—Analysis of motion using gradient-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds

Abstract

The invention discloses an RGBD visual inertia simultaneous positioning and map construction method, device, equipment and medium based on dotted line feature fusion, wherein the method comprises a front-end visual odometer, a rear-end optimization and a three-dimensional map construction process, images acquired by an RGBD camera and an IMU detection result are used as input information of an SLAM system in the front-end visual odometer process, the front-end visual odometer is carried out based on dotted line features and comprises feature detection and matching, IMU pre-integration and visual inertia alignment, and the feature detection and matching comprises point feature extraction and tracking and line feature extraction and tracking. The RGBD visual inertia simultaneous positioning and map construction method, device, equipment and medium based on dotted line feature fusion disclosed by the invention solve the problems of efficient state estimation and high-precision three-dimensional map construction of an autonomous robot under illumination change and low-texture indoor scenes, and have the advantages of high precision, high efficiency and the like.

Description

RGBD visual inertia simultaneous positioning and map construction with dotted line feature fusion
Technical Field
The invention relates to an RGBD visual inertia simultaneous localization and map construction system and method based on dotted line feature fusion, and belongs to the technical field of autonomous robot simultaneous localization and map construction.
Background
One of the goals of robots is autonomous operation in the real world, while the Localization and Mapping (SLAM) system is an autonomous robot key technology.
Efficient state estimation and accurate three-dimensional map construction technology of autonomous robots in indoor scenes with illumination changes and weak textures face huge challenges. On one hand, in an indoor scene with illumination change and weak texture, the visual inertia simultaneous localization and map creation (SLAM) method based on the feature point method has the defects that the localization precision is greatly reduced due to the difficulty in extracting a large number of effective feature points, and the complete failure of the system can be caused even by video, motion blur and the like with poor texture.
Most of traditional line feature detection methods adopt an EDLines algorithm as a tool for line feature detection, but when the algorithm faces a complex background or a noisy image, too many short line segment features are easily detected, and computing resources for line segment detection, description and matching are wasted, so that the positioning accuracy is obviously reduced. In addition, the algorithm usually has the condition that too many adjacent similar line segments and long line segments are easily segmented into a plurality of short line segments, so that the subsequent line segment matching task becomes complicated, and the uncertainty of the SLAM system is increased.
The construction of the three-dimensional dense point cloud map is very dependent on the accuracy of a pose estimation algorithm, and if the pose estimation is not accurate enough, the constructed three-dimensional dense point cloud map is easy to overlap and distort
For the reasons, the inventor conducts deep research on the existing autonomous robot state estimation and three-dimensional dense point cloud map construction method, and provides an RGBD visual inertia simultaneous localization and map construction system and method based on point-line feature fusion.
Disclosure of Invention
In order to overcome the problems, the inventor conducts intensive research to design an RGBD visual inertia simultaneous localization and map construction method based on dotted line feature fusion, which comprises a front-end visual odometer, a rear-end optimization and a three-dimensional map construction process, and further, in the front-end visual odometer process, input information of a visual inertia simultaneous localization and map creation system is detection values of an image and an IMU acquired by an RGBD camera.
Further, the front-end visual inertial odometer is based on dotted line features, including feature detection and matching, IMU pre-integration, and visual inertial alignment,
the feature detection and matching comprises the extraction and tracking of point features and the extraction and tracking of line features.
Further, the extracting and tracking of line features comprises the following sub-steps: s11, restraining the length of the line segment; s12, extracting line features through near line combination and short line splicing; and S13, tracking the line features by an optical flow method. Preferably, step S13 includes the following sub-steps: s131, matching anchor points; s132, matching points with lines; and S133, matching the lines. Preferably, in the front-end visual inertial odometer process, there is further the step of: s14, parameterizing point and line characteristics; and in the process of parameterizing the point and line characteristics, carrying out error constraint on the line characteristics.
Preferably, the IMU pre-integration value is taken as a constraint between two consecutive camera frame images.
Preferably, in the back-end optimization process, all state variables in the sliding window are optimized by minimizing the sum of all measurement residuals.
The invention also provides an RGBD visual inertia simultaneous positioning and map building device based on dotted line feature fusion, which comprises a front-end visual odometer module, a rear-end optimization module and a three-dimensional map building module,
the front-end visual inertial odometer module comprises a feature detection and matching sub-module, an IMU pre-integration sub-module and a visual inertial alignment sub-module, wherein the IMU pre-integration sub-module takes an IMU pre-integration value as a constraint between two frames of images, and the back-end optimization module optimizes all state variables in a sliding window by minimizing the sum of all measurement residuals.
The present invention also provides an electronic device, comprising: at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform any of the above methods.
The present invention also provides a computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any of the above methods.
The invention has the advantages that:
(1) according to the RGBD visual inertia simultaneous positioning and map construction system based on dotted line feature fusion, the problem that the correlation error of the front end data of the visual inertia odometer based on the feature points is large in a weak texture scene is solved through the introduction of line features, and the estimation precision of the motion between adjacent frame images of the weak texture scene is effectively improved; (2) according to the RGBD visual inertia simultaneous localization and map construction system based on the point-line feature fusion, the problem of low line segment extraction quality of the traditional algorithm is optimized while the rapid extraction is ensured by using the length suppression, near line combination and broken line splicing strategies, and the mismatching rate of the line features of the system is reduced; (3) according to the RGBD visual inertia simultaneous localization and map construction system based on dotted line feature fusion, the efficiency of a line feature tracking algorithm is obviously improved by using an optical flow method to track line features; (4) according to the RGBD visual inertia simultaneous localization and map construction system based on the point-line feature fusion, the high-precision dense point cloud map is constructed, and efficient state estimation and accurate three-dimensional point cloud construction can be achieved in indoor scenes with illumination changes and weak textures.
Drawings
FIG. 1 illustrates a flow diagram of a dotted line feature fused RGBD visual inertia simultaneous localization and mapping method in accordance with a preferred embodiment of the present invention;
FIG. 2 illustrates comparison results of detection effects before and after line segment length suppression in a dotted line feature fused RGBD visual inertia simultaneous localization and mapping method according to a preferred embodiment of the present invention;
FIG. 3 is a schematic diagram illustrating simultaneous localization and mapping of multiple proximate segments into a single segment in a dotted line feature fused RGBD visual inertia in accordance with a preferred embodiment of the present invention;
FIG. 4 is a diagram illustrating the joining of shorter segments into a long segment in a dotted line feature fused RGBD visual inertia simultaneous localization and mapping method according to a preferred embodiment of the present invention;
FIG. 5 illustrates RGBD visual inertia simultaneous localization and mapping method in I with dotted line feature fusion according to a preferred embodiment of the present invention1Middle match I0A point process schematic diagram in the frame image;
FIG. 6 is a diagram illustrating a matching point-to-line process in a dotted line feature fused RGBD visual inertia simultaneous localization and mapping method according to a preferred embodiment of the present invention;
FIG. 7 is a diagram illustrating a match line-to-line process in a dotted line feature fused RGBD visual inertia simultaneous localization and mapping method according to a preferred embodiment of the present invention;
FIG. 8 is a schematic diagram illustrating RGBD visual inertia simultaneous localization and mapping method line feature reprojection error in dotted line feature fusion according to a preferred embodiment of the present invention;
FIG. 9-a shows the detection result of the cafe line characteristics in example 1; FIG. 9-b shows the detection results of the coffee shop spot feature in example 1; FIG. 9-c shows the results of cafe line detection and tracking in example 1;
FIG. 10-a shows the results of detection of the household line 1 features in example 1; FIG. 10-b shows the results of home 1-point feature detection in example 1; FIG. 10-c shows the results of line detection and tracking for household 1 in example 1;
FIG. 11-a shows the results of family 2 line feature detection in example 1; FIG. 11-b shows the results of family 2-point feature detection in example 1; FIG. 11-c shows the results of line detection and tracking for household 2 in example 1;
FIG. 12-a shows the results of the detection of the characteristics of the corridor lines in example 1; FIG. 12-b shows the detection result of the feature of the corridor point in embodiment 1; FIG. 12-c shows the corridor alignment detection and tracking results in example 1;
FIG. 13-a shows the results in the corridor scenario of example 1 compared to the true trajectory (Ground route) and error; FIG. 13-b shows the results of the corridor scenario in example 1 compared to the real trajectory (Ground route) and error in a top view perspective; FIG. 13-c shows the results of the corridor scene in example 1 compared to the real trajectory (Ground route) and error in side view viewing angle;
FIG. 14 shows the results of example 2 and comparative examples 3 to 5 in scenario Lab 14; FIG. 15 shows a translation error box plot on scene Lab 1; FIG. 16 shows a diagram of a rotational error box on scene Lab 1;
fig. 17 shows the positional drift of embodiment 2 in the scene Lab 3; fig. 18 shows the positional drift of embodiment 2 in the scene Lab 6; fig. 19 shows three-dimensional maps obtained in comparative example 3 and example 2; fig. 20 shows three-dimensional maps obtained in comparative example 3 and example 2; fig. 21 shows three-dimensional maps obtained in comparative example 3 and example 2.
Detailed Description
The invention is explained in more detail below with reference to the figures and examples. The features and advantages of the present invention will become more apparent from the description.
The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. While the various aspects of the embodiments are presented in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.
Most of the traditional visual inertia simultaneous localization and map creation (SLAM) methods are based on point feature detection and description, however, in indoor scenes with illumination change and weak texture, due to the fact that a large number of effective feature points are difficult to extract, the localization accuracy is greatly reduced, and the complete failure of the system can be caused even by video with poor texture, motion blur and the like.
In view of the above, the inventor introduces line features and Inertial Measurement (IMU) data into the conventional SLAM method, the line feature detection is more robust than the point feature detection under the low texture condition, and the inertial measurement data can still perform good frequency motion under the condition of a small number of features in an image sequence, so that the SLAM result is still stable when the view angle of the robot changes significantly.
The RGBD visual inertia simultaneous localization and map construction method based on the dotted line feature fusion comprises a front-end visual odometer, a rear-end optimization and a three-dimensional map construction process.
In the process of front-end visual odometry, the input information of the visual inertia simultaneous positioning and map creation system is an image acquired by an RGBD camera and the detection value of an IMU.
The RGBD camera is a camera capable of simultaneously acquiring output RGB information and depth information, and is one of cameras commonly used for image recognition, and the IMU is a detection unit commonly used for a robot and is a sensor for measuring acceleration and motion angular velocity of the robot.
Further, the front-end visual inertial odometer is based on point features and line features, including feature detection and matching, IMU pre-integration, visual inertial alignment, as shown in fig. 1.
The feature detection and matching comprises the extraction and tracking of point features and the extraction and tracking of line features.
In the present invention, the method for extracting and tracking the point features is not particularly limited, and a point feature extraction and tracking method in the conventional SLAM may be adopted, for example, a Shi-Tomasi corner point is extracted as a feature point, a KLT optical flow method is adopted to track the feature point, and a point with a large difference is tracked and eliminated based on a backward optical flow method.
Wherein, the shit-Tomasi corner extraction adopts the paper J.Shi, C.tomasi, Good features to track, in: 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 1994, pp.593-600; the KLT optoflow method uses the paper B.D.Lucas and T.Kanade, iterative image registration technique with An application to stereo vision, in: the methods of Proceedings of the 7th International Joint Conference on Artificial Intelligence (IJCAI), IEEE, 1981, pp.24-28; the reversed flow method was performed using the paper Baker S, Matthews I.Lucas-kanade 20years on: a unification frame [ J ]. International journal of computing, 2004, 56 (3): 221-255.
Compared with the traditional SLAM system, the method has the advantages that the extraction and tracking of the line features are added in the front-end visual odometer, and the additional constraint is provided for the system by adding the line features, so that the accuracy of the system state estimation is improved.
In the prior art, line feature extraction methods include an LSD method, an FLD method, an EDlines method and the like, wherein the LSD method and the FLD method have slow detection speed and cannot meet the timeliness requirement of SLAM; although the EDlines method has high detection speed and high detection efficiency, when the EDlines method faces a low-texture scene or a noise image, a large number of short line features are always detected, the calculation resources for line detection, description and matching are wasted, and the positioning accuracy is obviously reduced.
In the SLAM system formed by fusing point and line features, because the line features are only used for providing additional constraint to improve the accuracy of state estimation, the EDLines method is improved in the invention, so that the EDLines method only detects the obvious line features in a scene while the original detection efficiency is maintained, and a more effective line feature detection result is obtained.
Specifically, in the present invention, the extraction and tracking of line features comprises the following sub-steps:
s11, restraining the length of the line segment; s12, extracting line features through near line combination and short line splicing; s13, tracking line features through an optical flow method; preferably, after step S12, there may be step S14, point, line feature parameterization.
Since the longer line segments are more stable and can be repeatedly detected in multiple frames, the positioning accuracy also increases with the increase of the number of the longer line segments, and in step S11, the line segments with the length lower than the threshold are deleted through the length suppression of the line segments, thereby effectively improving the quality of the extracted line segments. The results of comparison of the detection effects before and after the suppression of the segment length are shown in fig. 2.
Specifically, the segment length suppression is represented as:
Figure BDA0003204985380000051
wherein leniIndicating the length of the ith line segment, len, in the imageminThreshold value, W, representing the characteristic length of the shortest lineIIs the width of the image, HIHeight of image, symbol
Figure BDA0003204985380000052
Represents rounding up, eta is a proportionality coefficient, (x)sp，ysp) Coordinates of origin of line segment (x)ep，yep) Representing the coordinates of the end point of the line segment.
In the invention, the minimum value of the image width and the image height is taken as one of key indexes of threshold setting, so that the line length inhibition can adapt to different image sizes.
Preferably, the value of the proportionality coefficient eta is 0.01-0.2, more preferably 0.1, and through a large number of experiments, the inventor finds that the value of the proportionality coefficient eta can guarantee the quality of line segment detection to the maximum extent.
Considering that longer line segment features tend to come from a continuous region with a larger gradient where the detection accuracy is more reliable, it is necessary to merge adjacent line segments so that the resulting line segments are longer.
In step S12, a plurality of closer line segments are merged into one line segment, i.e., a near line merge, as shown in fig. 3, and
and splicing a plurality of shorter line segments into a long line segment, namely splicing short lines, as shown in fig. 4.
As shown in FIG. 3, in the present invention, Li、Lj、LkRepresenting segments to be spliced, their end points being respectively
Figure BDA0003204985380000061
Figure BDA0003204985380000062
As shown in FIG. 4, in the present invention, Lu、LvRepresenting two segments to be spliced, the end points of which are respectively
Figure BDA0003204985380000063
Preferably, in step S12, the method includes the following sub-steps:
s121, sequencing the line segments; s122, screening the line segments; s123, filtering the screened line segment group to obtain a merging candidate line segment set and a splicing candidate line segment set; and S124, merging or splicing the line segments.
In step S121, the line segments are sorted by segment length, preferably in descending order from long to short, to obtain a sequence of line segments { L }1，L2，..，LnIn which L is1The line segment with the longest line segment length is shown.
Since longer line segments tend to come from image regions with continuous strong gradients, L is the longest line segment1It is more reliable to start with.
Further, mixing L1The outer line segments are represented as a set of remnant line segments
L＝{L2，L3，...，Ln} (two)
Wherein n is the total number of line segments.
In step S122, the filtering includes angle filtering, which can be expressed as:
Figure BDA0003204985380000064
wherein L isαRepresenting candidate line segment groups, L, obtained by angle screeningmRepresenting different line segments, L, in the set of remnant line featuresmAngle in horizontal direction is thetam，θminIs an angle threshold for measuring the similarity of line segments.
The inventor determines theta through a large number of experimentsminIt is most preferable at π/90.
In step S123, the obtained line segment candidate group L is subjected toαAnd performing filtering, wherein the filtering of the near line merging process is represented as:
Figure BDA0003204985380000071
through the filtering process, a merging candidate line segment set can be obtained
Figure BDA0003204985380000074
The filtering of the stub splicing process is represented as:
Figure BDA0003204985380000072
through the filtering process, a splicing candidate line segment set can be obtained
Figure BDA0003204985380000073
In the formula IV and the formula V, dminAnd the threshold value represents the proximity of the measured line segment, and is preferably 5-15 pixels, and more preferably 9 pixels through a large amount of experimental analysis.
Through the filtering process, the angle and the space distance can be close to L1The line segment sets are grouped into a group to obtain a candidate line segment set Lβ＝{Lβ1，Lβ2}。
In step S124, the filtered candidate line segment set LβAdding line segment L1Form a new set of line segments { L }1，LβAnd selecting a head end point and a tail end point which are farthest from each other in the line segment set as a starting point and an ending point of a new line segment, and synthesizing a new line segment LM。
Recalculating the resultant line segment LMAngle thetaM，
If theta is satisfiedM＜θminThen set line segments { L }1，LβAre merged or spliced to form a composite line segment LMSet of replacement line segments { L1，Lβ}；
If not satisfy thetaM＜θminAnd then giving up merging or splicing, wherein the angle difference between the front and the rear of merging is overlarge, and the synthetic result deviates from the original line segment.
And repeating the process, and merging and splicing all line segments in the image.
The step S12 is one of the problems to be solved by the present invention, which is how to determine the correspondence between line segments in two consecutive images and further implement the tracking of the line segments.
In step S13, the points on the line segment are matched by the optical flow method, and then the matching points are counted to realize the tracking of the line segment.
Optical flow is commonly used for tracking point features, and is improved in the present invention to enable tracking of line segments.
Specifically, the method comprises the following substeps: s131, matching anchor points; s132, matching points with lines; and S133, matching the lines.
In step S131, the anchor points are spaced points in a line segment, and one line segment is characterized by a plurality of anchor points.
In the present invention, two consecutive frame images are represented as I0And I1Further, I0The set of line segments in the frame image is represented as:
L0 frame＝0li 0li＝(0pi，0qi)，0≤i≤M0Wherein M is0Is represented by0The total number of line segments in the frame image,0liis represented by0The ith line segment in the frame image,0pi，0qiis represented by0Two end points of the ith line segment in the frame image;
likewise, I1The set of line segments in the frame image is represented as:
L1 frame＝1lk|1lk＝(1pk，1qk)，0≤k≤M1Wherein M is1Is represented by1The total number of line segments in the frame image,1lkis represented by1The k-th line segment in the frame image,1pk，1qkis represented by1Two end points of the k-th line segment in the frame image.
For I0Each line segment in the frame image0li＝(0pi，0qi) The start point, the end point, and a plurality of points in the direction of the start point and the end point of the line segment are selected as anchor points, and as shown in fig. 5, the anchor points are set as0Qi＝0λi，j|0≤j＜Ni，0λi，jRepresenting the jth anchor point, N, on the ith line segmentiRepresents the total number of anchor points on the ith line segment, and is preferably 12.
Further, the vector of the starting point and the end point direction is:
0ai＝[0ai，0，0ai，1]T＝(0qi-0pi)/||0qi-0pi||。
further, in I1In the frame image, I can be obtained0Each anchor point in the frame image0λi，jIn the normal direction0niUpper matching point1λi，jThe normal direction is as follows:
0ni＝[-0ai，1，0ai，0]T
in step S122, in I1In the frame image, each matching point is obtained1λi，jTo different line segments1lkObtaining a distance set dkThe shortest distance in the distance set is denoted as dminThe line segment corresponding to the shortest distance is represented as1lclosett(i，j)As shown in fig. 6, in this example,
further, if dmin＜dthresholdThen the matching point is considered1λi，jBelong to line segment1lclosest(i，j)；
Otherwise, consider the matching point1λi，jNot belonging to line segments1lclosest(i，j)Judging the next anchor point;
wherein d isthresholdThe empirical threshold is preferably 1 pixel.
In step S133, as shown in FIG. 7, for I0Arbitrary line segment in frame image0li，0≤i＜M0Each anchor point of (1) at1All have unique matching points in the frame image1λi，jThe matching point falls on the line segment1lclosest(i，j)The above. For I0Line segments within a frame0liCalculate it at I1Number of anchor points matched in frame, set as ZiIs a line segment1lmatch(i)Number of anchor points matched, if Zi/Ni> R, definition0liAnd1 l match(i)0≤match(i)＜M1matched to each other, where R is an empirical threshold, preferably 0.6 pixels.
The inventors have found that although a solution based on line features is a good choice in low-texture scenes lacking point features, line feature occlusion or false detection tends to occur in repetitive scenes or low-texture scenes. In the present invention, the above problem is solved by a line feature error constraint.
In step S14, the point and line features are parameterized, and in the parameterization process, the line features are subjected to error constraint.
Specifically, in the present invention, the errors of the point feature and the line feature are constrained as follows:
Figure BDA0003204985380000091
in which, as shown in figure 8,
Figure BDA0003204985380000092
in the case of a point projection error term,
Figure BDA0003204985380000093
is a line projection error term; t isIwRepresenting the state of the RGBD camera in the I frame image in the world coordinate system, Xw，jDenotes the j-th point, x, observed in the I-th framei，jIs Xw，jProjection onto a plane; pw，k，Qw，kTwo end points, p, representing the ith line segment in the I-th framei，k，qi，kIs Pw，k，Qw，kThe point on the corresponding image plane is,
Figure BDA0003204985380000095
is Pw，k，Qw，kThe end points of the projection on the image plane,
Figure BDA0003204985380000099
is p in the world coordinate systemi，kTo the projected line segment
Figure BDA0003204985380000096
The distance of (a) to (b),
Figure BDA00032049853800000910
is q under the world coordinate systemi，kTo the projected line segment
Figure BDA0003204985380000097
The distance of (d); pi () for camera projection model, pih() Projecting the homogeneous coordinates of the model for the camera, further:
Figure BDA0003204985380000098
where O is the point coordinate in the RGBD camera coordinate system, PwAs point coordinates in the world coordinate system, RwRepresenting coordinate rotation, twRepresenting coordinate translation, fx，fyRepresenting the focal length of an RGBD camera, cx，cyRepresenting the principle points of the RGBD camera.
Further, Pw，kThe coordinate of the corresponding camera coordinate system is Op，Qw，kThe coordinate of the corresponding camera coordinate system is OqObtainable by the following formula:
Figure BDA0003204985380000101
then line segment
Figure BDA0003204985380000102
To the end point Pw，k，Qw，kThe Jacobian matrix of the reprojection errors of (1) is:
Figure BDA0003204985380000103
wherein the content of the first and second substances,
Figure BDA0003204985380000104
is OpComponents on the x, y, z coordinate axes of the camera coordinate system;
Figure BDA0003204985380000105
is OqComponents on the x, y, z coordinate axes of the camera coordinate system;
further, the air conditioner is provided with a fan,
Figure BDA0003204985380000106
lppointing to the optical center of a camera
Figure BDA0003204985380000107
Vector of (a) < i >qPointing to the optical center of a camera
Figure BDA0003204985380000108
The vector of (2).
In the conventional SLAM, a BA-optimization-based method is mostly adopted, an IMU pre-integration process is adopted, the frame speed and rotation need to be updated during each state adjustment, and re-integration is needed after each iteration, which makes the transmission strategy very time-consuming.
IMU measurements typically include three orthogonal axis accelerometers and three axis gyroscopes for measuring acceleration and angular velocity relative to the inertial frame. Affected by noise, IMU measurements typically contain an additive white noise n and a time-varying bias b, expressed as:
Figure BDA0003204985380000109
wherein, atRepresenting the ideal noise free value of three orthogonal axis accelerometers,
Figure BDA00032049853800001010
the offset of the acceleration is represented by the acceleration offset,
Figure BDA00032049853800001011
rotation g representing world coordinate system to body coordinate systemwRepresenting the acceleration of gravity in the world coordinate system, naIndicating accelerationMeter noise, omegatRepresenting an ideal value of the three-axis gyroscope without bias,
Figure BDA00032049853800001012
representing the gyroscope bias, nwRepresenting gyroscope noise.
Further, accelerometer noise naAnd gyroscope noise nwCan be expressed as a form of gaussian white noise:
Figure BDA0003204985380000111
further, the acceleration is biased
Figure BDA0003204985380000112
The derivative of (c) and the derivative of the gyroscope bias can be expressed in the form of gaussian white noise:
Figure BDA0003204985380000113
in the present invention, the IMU pre-integration value is used as a constraint between two frames of images, together with the camera re-projection residual, in the local window optimization process.
In particular, bk、bk+1For two successive image frames, bkThe time corresponding to a frame is tk，bk+1The time corresponding to a frame is tk+1，tkAnd tk+1The IMU pre-integration value is adopted to carry out the following constraint:
Figure BDA0003204985380000114
wherein the content of the first and second substances,
Figure BDA0003204985380000115
denotes bkFrame to bk+1The translation of the frame is carried out by a translation,
Figure BDA0003204985380000116
denotes bkFrame to bk+1The speed of the frame is such that,
Figure BDA0003204985380000117
denotes bkFrame to bk+1The rotation of the frame is carried out by rotating the frame,
Figure BDA0003204985380000118
representing τ Frames to bkThe rotation of the frame is carried out by rotating the frame,
Figure BDA0003204985380000119
denotes the rotation from time t to bk frame and τ denotes the intermediate variable of time during the double integration.
Further, Ω (·) can be expressed as:
Figure BDA00032049853800001110
ω ^ represents an antisymmetric matrix.
In the invention, when the pre-integration calculation is carried out between two continuous camera frames, the Jacobian matrix of each pre-integration term is calculated, and the effect of correcting the influence of bias change is achieved. Further, if the estimated deviation change is large, the pre-integration process is propagated again under the new deviation value, and through the process, a large amount of calculation is saved, the calculation efficiency is improved, and the positioning and map building speed is improved.
The visual inertial alignment adopts a loose coupling scheme of vision and IMU, firstly, a structural motion method (SFM) is utilized to estimate the camera attitude and the spatial position of a three-dimensional point, and then, the camera attitude and the spatial position are aligned with IMU pre-integration data to obtain the gravity direction, the scale factor, the gyroscope bias and the corresponding speed of each frame.
According to the present invention, in the back-end optimization process, unlike the conventional SLAM method, all state variables in the sliding window are optimized by minimizing the sum of all measurement residuals, and the process can be expressed as:
Figure BDA0003204985380000121
wherein the content of the first and second substances,
Figure BDA0003204985380000122
measuring residual error, χ, from frame I to frame I +1 in a body coordinate systembFor the set of all IMU pre-integrals within the sliding window,
Figure BDA0003204985380000123
for the re-projection error of the point feature,
Figure BDA0003204985380000124
reprojection error, x, for line featurespFor the set of point features observed in frame I, xlFor the set of line features observed in frame I, epriorFor a priori information, ρ () is a Cauchy function, which plays a role in suppressing an outlier.
In the three-dimensional mapping process, as in the conventional SLAM method, local point clouds are spliced into a final point cloud map by means of estimated camera poses.
Accurate pose estimation plays an important role in the accuracy of the point cloud picture. If the pose estimate is not accurate enough, the final point cloud images may overlap or bow. In the invention, as the feature tracking node is adopted to track the features on the RGB image and obtain the distance information from the depth map, and another scale information source is obtained by utilizing the depth map, the robustness and the accuracy of the system are improved, and the reconstructed three-dimensional cloud map is more reliable.
Furthermore, in the invention, three-dimensional points are projected from each depth image, and are synchronous with the key frame, the corresponding postures are estimated, and the cloud picture is constructed, so that the system is compact and is more suitable for small-sized robot integration. Moreover, the point cloud picture of the type can be directly used for semantic segmentation or recognition and further used for obstacle avoidance and path planning.
On the other hand, the invention also discloses a device which comprises a front-end visual odometer module, a rear-end optimization module and a three-dimensional drawing building module.
The front-end visual inertial odometer module comprises a feature detection and matching sub-module, an IMU pre-integration sub-module and a visual inertial alignment sub-module.
The feature detection and matching submodule is used for extracting and tracking point features and extracting and tracking line features.
Further, in the feature detection and matching submodule, the line feature is extracted and tracked through steps S11 to S13.
The IMU pre-integration submodule takes an IMU pre-integration value as a constraint between two frames of images, and the constraint is used together with a camera re-projection residual error in the local window optimization process, and the constraint is expressed as follows:
Figure BDA0003204985380000131
the visual inertial alignment submodule estimates the camera attitude and the spatial position of a three-dimensional point by using a structure motion method (SFM), and then aligns the camera attitude and the spatial position with IMU pre-integration data to obtain the gravity direction, the scale factor, the gyroscope bias and the corresponding speed of each frame.
The back-end optimization module optimizes all state variables in the sliding window by minimizing the sum of all measurement residuals, and the process can be expressed as:
Figure BDA0003204985380000132
and the three-dimensional mapping module is used for splicing the local point clouds into a final point cloud map by means of the estimated camera attitude.
Various embodiments of the above-described methods and apparatus of the present invention may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), System On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for implementing the methods of the present invention may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of the present invention, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the methods and apparatus described herein may be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The methods and apparatus described herein may be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The Server may be a cloud Server, which is also called a cloud computing Server or a cloud host, and is a host product in a cloud computing service system, so as to solve the defects of large management difficulty and weak service extensibility in the conventional physical host and VPS service ("Virtual Private Server", or VPS "for short). The server may also be a server of a distributed system, or a server incorporating a blockchain.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in this disclosure may be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed herein can be achieved, and the present disclosure is not limited herein.
Examples
Example 1
The simulation experiment of inertia simultaneous localization and map creation is carried out by adopting an open data set, wherein the data set is OpenLORIS-Scene, and is collected in 5 scenes of an office, a corridor, a family, a cafe and a market, which are very typical autonomous robot application scenes.
The simultaneous localization of the inertia and the map creation comprises a front-end visual odometer, a rear-end optimization and a three-dimensional map building process.
Extracting and tracking point features and line features in the process of front-end visual odometry, wherein when the point features are extracted and tracked, Shi-Tomasi angular points are extracted as feature points, a KLT optical flow method is adopted to realize the tracking of the feature points, and points with larger differences are tracked and eliminated based on a reverse optical flow method;
the line feature extraction tracking comprises the following sub-steps:
s11, restraining the length of the line segment; s12, extracting line features through near line combination and short line splicing; s13, tracking line features through an optical flow method; and S14, parameterizing point and line characteristics.
In step S11, the segment length suppression is expressed as:
Figure BDA0003204985380000151
eta is 0.1.
Step S12 includes the following substeps: s121, sequencing the line segments; s122, screening the line segments; s123, filtering the screened line segment group to obtain a merging candidate line segment set and a splicing candidate line segment set; and S124, merging or splicing the line segments.
In step S121, the line segments are sorted in the order of length of the line segment from long to short, and in step S122, the screening is an angle screening, which is expressed as:
Figure BDA0003204985380000152
wherein, thetaminIs pi/90.
In step S123, the obtained line segment candidate group L is subjected toαAnd performing filtering, wherein the filtering of the near line merging process is represented as:
Figure BDA0003204985380000154
the filtering of the stub splicing process is represented as:
Figure BDA0003204985380000162
dmina threshold value representing the proximity of the measured line segment, whose value is 9 pixels.
Step S13 includes the following substeps: s131, matching anchor points; s132, matching points with lines; and S133, matching the lines.
Wherein, the total number of anchor points in the substep S131 is 12.
In step S14, parameterizing the point and line features, and in the parameterizing process, performing error constraint on the line features, and constraining the errors of the point and line features as follows:
Figure BDA0003204985380000163
in the process of front-end visual odometry, an IMU pre-integration value is used as a constraint between two frames of images, and is used together with a camera re-projection residual error in the process of local window optimization, so that the following constraints are performed:
Figure BDA0003204985380000164
in the process of the front-end visual odometer, a loose coupling scheme of vision and an IMU is adopted for visual inertial alignment, the camera attitude and the spatial position of a three-dimensional point are estimated by using a structure motion method (SFM), and then the camera attitude and the spatial position are aligned with IMU pre-integration data to obtain the gravity direction, the scale factor, the gyroscope bias and the corresponding speed of each frame.
In the back-end optimization process, all state variables in the sliding window are optimized by minimizing the sum of all measurement residuals, which process can be expressed as:
Figure BDA0003204985380000165
in the three-dimensional mapping process, the local point clouds are spliced into a final point cloud map by means of the estimated camera pose.
Fig. 9 to 12 are point and line feature detections in a plurality of low texture and illumination variation scenes in an OpenLORIS-Scene data set, where fig. 9 is a coffee shop detection result, fig. 10 is a home 1 detection result, fig. 11 is a home 2 detection result, and fig. 12 is a corridor detection result; FIGS. 9-a, 10-b, 11-c, 12-d show results of line feature detection, and FIGS. 9-a, 10-b, 11-c, 12-d show results of point feature detection.
It can be seen that the indoor scene is mostly an artificial structure scene, has linear structures such as abundant edge, right angle, changes the low texture scene (coffee shop, corridor) that is stronger in the illumination, and the characteristic is not abundant (family 1, family 2), and the point feature detection is not enough, but, line feature detection algorithm still can obtain abundant line characteristic. Compared with point detection, the point-line feature complementation method can provide richer and more robust feature information for subsequent motion estimation.
9-a, 10-b, 11-c, 12-d show the line detection and tracking in different scenarios, more effective information can be obtained by using a line feature matching algorithm, the geometric structure can be described more intuitively, and richer and more robust information is provided for the subsequent visual range to estimate the camera motion according to the comprehensive features.
Fig. 13-a is a comparison of the results in a corridor scene with the true trajectory (Ground route) and error, where the color changes from blue to red to indicate a gradual increase in error, and fig. 13-b, 13-c are the results in a top view viewing angle and a side view viewing angle.
As the speed of motion increases, the viewing angle changes significantly due to the introduction of more motion blur, the illumination conditions are also challenging to track features, and performance may degrade. As can be seen from the figure, the present embodiment can accurately align with the real trajectory, and especially when the camera rotates rapidly, drift free tracking can be realized, which means that adding depth information to the visual inertial system has great improvement on pose estimation.
Example 2
And carrying out a positioning and mapping experiment in an indoor environment, and verifying the positioning accuracy according to the data of the motion capture system OptiTrack.
The experimental move was a drone equipped with an intel RealSense D435i depth camera, intel NUC onboard computer, and Pixhawk V5+ flight controls. The D435i depth camera, which incorporates a six-axis IMU sensor, can capture 640 x 800 resolution color images. In the experiment, the sampling frequencies of the camera and IMU were set to 30Hz and 200Hz, respectively. And the onboard computer processes the image and IMU data acquired by the depth inertial camera in real time.
14 different scenes (Lab 1-Lab 14) are built indoors, and the 14 scenes respectively have the characteristics of low texture, illumination change, rapid acceleration and angular rotation.
And the on-board computer positions the pictures collected by the depth camera to map according to the method in the embodiment 1.
Comparative example 1
The same experiment as in example 1 was conducted, except that the test was conducted in a paper R.Mur-Artal, J.D.Tard. Louis, ORB-SLAM 2: the ORB-SLAM2 method in Anopen-source SLAMS for monocula, stereo, and RGB-Dcameras, IEEE Transactions on Robotics 33(5) (2017)1255-1262, ORB-SLAM2 is a point-based monocular SLAM system that can operate in large, small, indoor and outdoor environments by adding RGBD interfaces.
Comparative example 2
The same experiment as in example 1 was performed, except that the following paper q.fu, j.wang, h.yu, i.ali, f.guo, y.he, h.zhang, PL-VINS: real time monoclonal visual-interactive SLAMS with points and line features (2020). 2009.07462. The PL-VINS method is a monocular vision inertial system optimization method with point and line characteristics developed on the basis of point-based VINS-M0 no.
Comparative example 3
The same experiment as in example 2 was carried out, except that the VINS-RGBD method in article Z.Shan, R.Li, S.Schwertfeger, RGBD-inert project estimation and mapping for ground loops, Sensors 19(10) (2019)2251 was used.
Comparative example 4
The same experiment as in example 2 was performed, except that the following paper, R.Mur-Artal, J.D.Tard. Louis, ORB-SLAM 2: the ORB-SLAM2 method in open-source SLAMS for monoclonal, stereo, and RGB-Dcameras, IEEE Transactions on Robotics 33(5) (2017)1255-1262.
Comparative example 5
The same experiment as in example 2 was performed, except that the following paper q.fu, j.wang, h.yu, i.ali, f.guo, y.he, h.zhang, PL-VINS: real time monoclonal visual-interactive SLAMS with points and line features (2020). 2009.07462. The PL-VINS method of (1) is carried out.
Examples of the experiments
Experimental example 1
The results of the experiments comparing example 1 with comparative examples 1 and 2 were evaluated using Root Mean Square Error (RMSE), maximum error (max error) and mean error (mean error), and are shown in table one.
Watch 1
Figure BDA0003204985380000191
As can be seen from the table I, the RMSE in the embodiment 1 is obviously lower than that in the scenes of the comparative example 1, the marker 1-1 and the corridor1-4, the tracking fault occurs in the comparative example 2, the serious drift occurs, and the final track cannot be obtained, and the simulation result shows that the embodiment 1 has better robustness.
In addition, in the case of fast motion, i.e. in the sparse feature sequence Office 1-7, embodiment 1 can still maintain high-precision positioning of about 0.18m, which is about 44% higher than that of comparative example 1.
And the positioning error fluctuation of the embodiment 1 is small, the consistency of the positioning estimation is good, and the robust accurate positioning is realized in the low-texture environment.
Experimental example 2
The RMSE (unit: m) results of comparative example 2 and comparative examples 3 to 5 are shown in Table two.
Watch two
Figure BDA0003204985380000201
As can be seen from table two, embodiment 1 is superior in low texture and accuracy of illumination variation in the room, and embodiment 1 has minimal overall fluctuation, high stability, and root mean square error within 0.44 meters.
The results of example 2 and comparative examples 3 to 5 in the Lab14 are shown in FIG. 14, and it can be seen from the graph that the positioning track of example 2 is closer to OptiTrack, and the positioning error is significantly smaller than that of comparative examples 3 to 5.
Fig. 15 shows a translational error box diagram on the scene Lab1, and fig. 16 shows a rotational error box diagram on the scene Lab 1. As can be seen from the figure, the upper and lower concentration distributions of example 2 are smaller than those of comparative examples 3 to 5.
Fig. 17 shows the position drift of the embodiment 2 in the scene Lab3, and it can be seen from the figure that the triaxial error is concentrated within 300mm, and fig. 18 shows the position drift of the embodiment 2 in the scene Lab6, and it can be seen from the figure that the triaxial error is concentrated within 200mm, which illustrates that the change of the embodiment 2 is smooth, and the positioning accuracy and stability are good.
Fig. 19 to 21 show three-dimensional maps obtained in comparative example 3 and example 2, wherein the left column is the result obtained in comparative example 3, and the right column is the result obtained in example 2, and it can be seen from the figures that comparative example 3 has a large error in the horizontal and vertical directions, double shading and bending phenomena occur during the splicing process, and the map construction precision is low. In embodiment 2, the map stitching effect is greatly improved, and the problems of double shadows and curvature are solved, which also means that the accumulated error in the map construction process is reduced, and embodiment 2 can well complete the construction of the indoor map.
The present invention has been described above in connection with preferred embodiments, but these embodiments are merely exemplary and merely illustrative. On the basis of the above, the invention can be subjected to various substitutions and modifications, and the substitutions and the modifications are all within the protection scope of the invention.

Claims (10)

1. An RGBD visual inertia simultaneous localization and map construction method based on dotted line feature fusion comprises a front-end visual odometer, a rear-end optimization and a three-dimensional map construction process, and is characterized in that,
in the process of front-end visual odometry, the input information of the visual inertia simultaneous positioning and map creation system is an image acquired by an RGBD camera and the detection value of an IMU.
2. The RGBD visual inertia simultaneous localization and mapping method based on dotted line feature fusion according to claim 1,
the front-end visual inertial odometer is based on dotted line features, including feature detection and matching, IMU pre-integration, and visual inertial alignment,
the feature detection and matching comprises the extraction and tracking of point features and the extraction and tracking of line features.
3. The RGBD visual inertia simultaneous localization and mapping method based on dotted line feature fusion according to claim 2,
the extraction and tracking of line features comprises the following sub-steps:
s11, restraining the length of the line segment;
s12, extracting line features through near line combination and short line splicing;
and S13, tracking the line features by an optical flow method.
4. The RGBD visual inertia simultaneous localization and mapping method based on dotted line feature fusion according to claim 3,
step S13 includes the following substeps:
s131, matching anchor points;
s132, matching points with lines;
and S133, matching the lines.
5. The RGBD visual inertia simultaneous localization and mapping method based on dotted line feature fusion according to claim 2,
in the front-end visual inertial odometer process, the method also comprises the following steps:
s14, parameterizing point and line characteristics;
and in the process of parameterizing the point and line characteristics, carrying out error constraint on the line characteristics.
6. The RGBD visual inertia simultaneous localization and mapping method based on dotted line feature fusion according to claim 2,
the IMU pre-integration value is taken as a constraint between two consecutive camera frame images.
7. The RGBD visual inertia simultaneous localization and mapping method based on dotted line feature fusion according to claim 2,
in the back-end optimization process, all state variables in the sliding window are optimized by minimizing the sum of all measurement residuals.
8. An RGBD visual inertia simultaneous positioning and map construction device based on dotted line feature fusion is characterized by comprising a front-end visual odometer module, a rear-end optimization module and a three-dimensional map construction module,
the front-end visual inertial odometer module comprises a feature detection and matching sub-module, an IMU pre-integration sub-module and a visual inertial alignment sub-module,
the IMU pre-integration sub-module takes the IMU pre-integration value as the constraint between two frames of images,
the back-end optimization module optimizes all state variables in the sliding window by minimizing the sum of all measured residuals.
9. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.
10. A computer-readable storage medium having computer instructions stored thereon for causing the computer to perform the method of any one of claims 1-7.
CN202110914560.8A 2021-08-10 2021-08-10 RGBD visual inertia simultaneous positioning and map construction with dotted line feature fusion Pending CN113763470A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110914560.8A CN113763470A (en) 2021-08-10 2021-08-10 RGBD visual inertia simultaneous positioning and map construction with dotted line feature fusion

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110914560.8A CN113763470A (en) 2021-08-10 2021-08-10 RGBD visual inertia simultaneous positioning and map construction with dotted line feature fusion

Publications (1)

Publication Number Publication Date
CN113763470A true CN113763470A (en) 2021-12-07

Family

ID=78789041

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110914560.8A Pending CN113763470A (en) 2021-08-10 2021-08-10 RGBD visual inertia simultaneous positioning and map construction with dotted line feature fusion

Country Status (1)

Country Link
CN (1) CN113763470A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117197211A (en) * 2023-09-04 2023-12-08 北京斯年智驾科技有限公司 Depth image generation method, system, device and medium

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117197211A (en) * 2023-09-04 2023-12-08 北京斯年智驾科技有限公司 Depth image generation method, system, device and medium
CN117197211B (en) * 2023-09-04 2024-04-26 北京斯年智驾科技有限公司 Depth image generation method, system, device and medium

Similar Documents

Publication Publication Date Title
Dai et al. Rgb-d slam in dynamic environments using point correlations
CN109166149B (en) Positioning and three-dimensional line frame structure reconstruction method and system integrating binocular camera and IMU
US11461912B2 (en) Gaussian mixture models for temporal depth fusion
US9613420B2 (en) Method for locating a camera and for 3D reconstruction in a partially known environment
WO2022188094A1 (en) Point cloud matching method and apparatus, navigation method and device, positioning method, and laser radar
CN109307508A (en) A kind of panorama inertial navigation SLAM method based on more key frames
Liu et al. Towards robust visual odometry with a multi-camera system
Kim et al. Direct semi-dense SLAM for rolling shutter cameras
Hwangbo et al. Visual-inertial UAV attitude estimation using urban scene regularities
Krombach et al. Feature-based visual odometry prior for real-time semi-dense stereo SLAM
CN112802196B (en) Binocular inertia simultaneous positioning and map construction method based on dotted line feature fusion
CN112556719B (en) Visual inertial odometer implementation method based on CNN-EKF
Koch et al. Wide-area egomotion estimation from known 3d structure
CN114485640A (en) Monocular vision inertia synchronous positioning and mapping method and system based on point-line characteristics
CN112945233B (en) Global drift-free autonomous robot simultaneous positioning and map construction method
CN113763470A (en) RGBD visual inertia simultaneous positioning and map construction with dotted line feature fusion
Zhu et al. PairCon-SLAM: Distributed, online, and real-time RGBD-SLAM in large scenarios
CN112731503A (en) Pose estimation method and system based on front-end tight coupling
Yang et al. PLS-VINS: Visual inertial state estimator with point-line features fusion and structural constraints
Cigla et al. Gaussian mixture models for temporal depth fusion
Vlaminck et al. A markerless 3D tracking approach for augmented reality applications
Zhao et al. A review of visual SLAM for dynamic objects
CN116147618A (en) Real-time state sensing method and system suitable for dynamic environment
CN112991400B (en) Multi-sensor auxiliary positioning method for unmanned ship
Laskar et al. Robust loop closures for scene reconstruction by combining odometry and visual correspondences

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination