CN110349250A - A kind of three-dimensional rebuilding method of the indoor dynamic scene based on RGBD camera - Google Patents
A kind of three-dimensional rebuilding method of the indoor dynamic scene based on RGBD camera Download PDFInfo
- Publication number
- CN110349250A CN110349250A CN201910572096.1A CN201910572096A CN110349250A CN 110349250 A CN110349250 A CN 110349250A CN 201910572096 A CN201910572096 A CN 201910572096A CN 110349250 A CN110349250 A CN 110349250A
- Authority
- CN
- China
- Prior art keywords
- dynamic
- point
- frame
- camera
- convolutional neural
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 51
- 238000001514 detection method Methods 0.000 claims abstract description 21
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 20
- 238000005457 optimization Methods 0.000 claims abstract description 9
- 238000003780 insertion Methods 0.000 claims abstract description 5
- 230000037431 insertion Effects 0.000 claims abstract description 5
- 230000006872 improvement Effects 0.000 claims description 8
- 230000011218 segmentation Effects 0.000 claims description 5
- 241001465754 Metazoa Species 0.000 claims description 4
- 238000012937 correction Methods 0.000 claims description 4
- 239000000284 extract Substances 0.000 claims description 4
- 239000011159 matrix material Substances 0.000 claims description 4
- 238000012546 transfer Methods 0.000 claims description 4
- 230000009466 transformation Effects 0.000 claims description 4
- 238000003709 image segmentation Methods 0.000 claims description 3
- 238000005259 measurement Methods 0.000 claims 2
- 238000007689 inspection Methods 0.000 claims 1
- 238000007500 overflow downdraw method Methods 0.000 claims 1
- 238000012549 training Methods 0.000 claims 1
- 238000004804 winding Methods 0.000 claims 1
- 230000008030 elimination Effects 0.000 abstract description 5
- 238000003379 elimination reaction Methods 0.000 abstract description 5
- 230000000694 effects Effects 0.000 abstract description 3
- 238000013507 mapping Methods 0.000 description 6
- 230000003068 static effect Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 230000004927 fusion Effects 0.000 description 3
- 238000010276 construction Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 241000282472 Canis lupus familiaris Species 0.000 description 1
- 241000282326 Felis catus Species 0.000 description 1
- 206010047571 Visual impairment Diseases 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000019771 cognition Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000003238 somatosensory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000010408 sweeping Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Software Systems (AREA)
- General Health & Medical Sciences (AREA)
- Mathematical Physics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biophysics (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Biomedical Technology (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Quality & Reliability (AREA)
- Health & Medical Sciences (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Image Analysis (AREA)
Abstract
本发明公开了一种基于RGBD相机的室内动态场景的三维重建方法,包括标定RGBD相机、采集场景图像、提取特征点、动态点检测与剔除、利用卷积神经网络和多视图几何法相结合的方式来进行动态点的剔除、再跟踪、插入新关键帧、局部地图优化、回环检测等步骤,使用该方法有效的解决了三维重建在动态场景下相机位姿估计不准确以及动态物体重影的问题,本发明对动态区域检测步骤进行了时间上的优化,可以实现实时重建的效果。在TUM的数据集上我们用相机轨迹绝对误差进行衡量,其均方根误差由先前的0.752提高至0.025。
The invention discloses a three-dimensional reconstruction method of an indoor dynamic scene based on an RGBD camera, including the method of calibrating the RGBD camera, collecting scene images, extracting feature points, detecting and eliminating dynamic points, and combining convolutional neural network and multi-view geometric method To perform steps such as dynamic point elimination, re-tracking, insertion of new key frames, local map optimization, and loop detection. Using this method effectively solves the problem of inaccurate camera pose estimation and dynamic object ghosting in 3D reconstruction in dynamic scenes. , the present invention optimizes the time of the dynamic area detection step, and can realize the effect of real-time reconstruction. On the TUM dataset, we use the absolute error of the camera trajectory to measure, and its root mean square error has increased from the previous 0.752 to 0.025.
Description
技术领域technical field
本发明属于三维成像领域,具体的说,是一种基于RGBD相机的室内动态场景的三维重建方法。The invention belongs to the field of three-dimensional imaging, in particular, it is a three-dimensional reconstruction method of an indoor dynamic scene based on an RGBD camera.
背景技术Background technique
在近些年的计算机技术的进步,AR、VR技术逐渐成为研究的热点领域之一,在智能家居领域,在虚拟购物领域有着重要的应用,如何有效的重建周围的场景是其中的研究的方向之一。随着人工智能技术的进步,在汽车自动驾驶领域,无人机自主飞行领域同样需要解决周围场景的构建以及自身的定位问题。With the advancement of computer technology in recent years, AR and VR technology has gradually become one of the hot research areas. In the field of smart home, it has important applications in the field of virtual shopping. How to effectively reconstruct the surrounding scenes is the direction of research. one. With the advancement of artificial intelligence technology, in the field of automatic driving of cars, the field of autonomous flight of drones also needs to solve the construction of surrounding scenes and its own positioning.
SLAM(同时定位与地图构建)技术的出现很好的解决了以上的问题。SLAM系统利用搭载于物体上传感器(例如视觉SLAM中的单目、双目相机)来获得外界信息,利用这些信息对自身的位置和姿态进行估计,根据需要构建出地图。The emergence of SLAM (simultaneous localization and map construction) technology has solved the above problems well. The SLAM system uses sensors mounted on objects (such as monocular and binocular cameras in visual SLAM) to obtain external information, uses this information to estimate its own position and attitude, and constructs maps as needed.
与此同时,近几年深度学习的神经网络在图像识别和分割领域的表现令人惊艳,Alex-Net,Google-Net,VGG,Res-Net等网络不断刷新着目标识别数据集的准确率。因此与深度学习相结合的语义SLAM系统有了新的发展。语义SLAM系统在对周围场景进行重建的同时能够对场景中的物体进行识别,将其应用在机器人领域可以提高机器人对周围场景的理解与认知,可以为机器人完成更加复杂的任务提供可能。将其应用在自动驾驶领域可以实现主动避障,危险预警等功能,因此语义SLAM有着广阔的应用前景。At the same time, in recent years, the performance of deep learning neural networks in the field of image recognition and segmentation has been amazing. Alex-Net, Google-Net, VGG, Res-Net and other networks have continuously refreshed the accuracy of target recognition datasets. Therefore, the semantic SLAM system combined with deep learning has a new development. The semantic SLAM system can recognize objects in the scene while reconstructing the surrounding scene. Its application in the field of robotics can improve the robot's understanding and cognition of the surrounding scene, and can provide the possibility for the robot to complete more complex tasks. Applying it in the field of automatic driving can realize active obstacle avoidance, danger warning and other functions, so semantic SLAM has broad application prospects.
传统的SLAM方法很好的解决了静态场景下的定位和建图问题,但是在有动态物体的场景中,这些方法的定位和建图的准确度很差,因为传统的SLAM方法难以区分出动态物体和静态物体,而它们进行相同的处理。但实际上,动态物体提取的特征点与静态背景的特征点运动时不一致的,这会严重影响相机位置的定位,从而影响建图的结果。在实际应用中,消除动态物体也是很有必要的,例如在扫地机器人的路径规划和导航中,如果不能消除人,狗,猫等动态物体,会使机器人的导航路径产生偏差。Traditional SLAM methods solve the positioning and mapping problems in static scenes very well, but in scenes with dynamic objects, the positioning and mapping accuracy of these methods is very poor, because traditional SLAM methods are difficult to distinguish dynamic objects. objects and static objects, and they are treated the same. But in fact, the feature points extracted by the dynamic object are inconsistent with the feature points of the static background, which will seriously affect the positioning of the camera position, thereby affecting the result of the mapping. In practical applications, it is also necessary to eliminate dynamic objects. For example, in the path planning and navigation of sweeping robots, if dynamic objects such as people, dogs, and cats cannot be eliminated, the robot's navigation path will be deviated.
本发明解决了SLAM系统中在动态场景下轨迹估计不准确,地图重建时产生重影的问题。本发明在SLAM系统中加入了卷积神经网络,利用卷积神经网络对物体进行分割,利用先验知识和多视图几何法对动态点进行判断,从而消除了动态点对场景的影响。The invention solves the problems of inaccurate trajectory estimation and ghosting during map reconstruction in the SLAM system. The present invention adds a convolutional neural network to the SLAM system, uses the convolutional neural network to segment objects, and uses prior knowledge and multi-view geometry to judge dynamic points, thereby eliminating the impact of dynamic points on the scene.
发明内容Contents of the invention
本发明的目的是针对现有三维重建方法在动态场景下存在的问题提出了一种基于RGBD相机和卷积神经网络的三维重建方法,是利用深度相机和彩色相机对室内场景进行三维重建,利用语义分割的方法消除了动态物体对最终重建效果的影响。The purpose of the present invention is to propose a three-dimensional reconstruction method based on RGBD cameras and convolutional neural networks for the problems existing in existing three-dimensional reconstruction methods in dynamic scenes. It uses depth cameras and color cameras to perform three-dimensional reconstruction of indoor scenes. The method of semantic segmentation eliminates the impact of dynamic objects on the final reconstruction effect.
本发明是通过以下技术方案来实现的:The present invention is achieved through the following technical solutions:
本发明公开了一种基于RGBD相机和卷积神经网络的三维重建方法,三维重建方法所用的装置主要包括RGBD相机,带有GPU的PC机,具体步骤如下:The invention discloses a three-dimensional reconstruction method based on an RGBD camera and a convolutional neural network. The devices used in the three-dimensional reconstruction method mainly include an RGBD camera and a PC with a GPU. The specific steps are as follows:
1)、标定RGBD相机:得到RGBD相机的彩色相机内参值和深度相机的内参值以及深度相机和彩色相机的转移矩阵;1), Calibrate the RGBD camera: Obtain the internal parameter value of the color camera of the RGBD camera and the internal parameter value of the depth camera, as well as the transfer matrix of the depth camera and the color camera;
2)、采集场景图像:每一帧包括一张彩色图和深度图,利用RGBD相机的SDK将彩色图与深度图对齐;2) Collect scene images: each frame includes a color map and a depth map, and use the SDK of the RGBD camera to align the color map with the depth map;
3)、提取特征点:利用ORB特征点算法提取出彩色图像中的特征点;3), extract feature points: use the ORB feature point algorithm to extract feature points in the color image;
4)、动态点检测与剔除:利用卷积神经网络和多视图几何法相结合的方式来进行动态点的剔除;4), dynamic point detection and elimination: use the combination of convolutional neural network and multi-view geometric method to eliminate dynamic points;
5)、再跟踪:在利用多视图几何法进一步剔除动态点后,重新利用速度模型和参考帧模型进行对当前帧位姿进行估计,在得到初始位姿后在局部地图中进行更精确的跟踪;5) Re-tracking: After using the multi-view geometry method to further eliminate dynamic points, re-use the velocity model and the reference frame model to estimate the pose of the current frame, and perform more accurate tracking in the local map after obtaining the initial pose ;
6)、插入新关键帧:判断插入新关键帧的依据是,距离上一次插入关键帧超过了20帧以上,当前帧追踪到了到的地图点少于50个,当前帧与参考帧的重合比例少于90%;6) Insert a new key frame: The basis for judging the insertion of a new key frame is that the distance from the last inserted key frame exceeds 20 frames, the number of map points tracked by the current frame is less than 50, and the overlap ratio between the current frame and the reference frame less than 90%;
7)、局部地图优化:对新加入的关键帧、新加入关键帧的所有共视帧及所有共视帧中的地图点进行BA优化;7), local map optimization: perform BA optimization on newly added key frames, all common view frames of newly added key frames, and map points in all common view frames;
8)、回环检测:利用当前关键帧的词袋模型判断是否与之前帧构成闭环,若过闭环一致性检,则计算闭环帧之间的相似变换,相邻帧进行闭环矫正。8) Loop closure detection: Use the bag-of-words model of the current key frame to determine whether it forms a closed loop with the previous frame. If it passes the closed-loop consistency check, calculate the similarity transformation between closed-loop frames, and perform closed-loop correction on adjacent frames.
作为进一步地改进,本发明所述的步骤4)中,动态点检测与剔除包括如下步骤:As a further improvement, in step 4) of the present invention, dynamic point detection and elimination include the following steps:
a)、图像分割与识别:利用重新训练的SegNet网络(一种基于卷积神经网络的语义分割网络)对彩色图像进行分割,得到每个像素的标签,得到的人和其他动物的区域属于动态区域,其内的特征点属于动态点,对这些动态点进行剔除操作;a), image segmentation and recognition: use the retrained SegNet network (a semantic segmentation network based on convolutional neural network) to segment the color image, get the label of each pixel, and the obtained areas of people and other animals are dynamic The feature points in the area belong to dynamic points, and these dynamic points are eliminated;
b)、跟踪:将当前帧剔除后的特征点和上一帧或者上一个参考帧做特征点匹配,将利用速度模型或参考帧模型估计出当前帧的一个初始位姿;b) Tracking: match the feature points of the current frame with the feature points of the previous frame or the previous reference frame, and use the velocity model or the reference frame model to estimate an initial pose of the current frame;
c)、多视图几何法判断动态点:通过比较某一空间特征点投影到当前帧上深度估计值与该点的深度测量值来进行判断,当其差值大于一定阈值是认为该点为动态点;c) Judgment of dynamic points by multi-view geometry method: Judgment is made by comparing the estimated depth value of a certain spatial feature point projected onto the current frame with the measured depth value of the point. When the difference is greater than a certain threshold, the point is considered to be dynamic. point;
d)、对多视图几何法得到动态点,对其在深度图上进行区域生长操作,得到动态区域;d) Obtain dynamic points from the multi-view geometry method, and perform region growing operations on the depth map to obtain dynamic regions;
e)、将卷积神经网络和多视图几何法得到的两个动态区域进行融合,融合的具体操作方法是对两个区域取并集。e) Fusing the two dynamic regions obtained by the convolutional neural network and the multi-view geometry method, the specific operation method of the fusion is to take the union of the two regions.
f)、作为进一步的改进,为了提高检测速度,每5帧进行一次动态区域检测,即每5帧重复以上a)-e)步骤。f) As a further improvement, in order to increase the detection speed, the dynamic region detection is performed every 5 frames, that is, the above steps a)-e) are repeated every 5 frames.
作为进一步地改进,本发明所述的步骤c)中的具体判断的原理如下:将空间点X的坐标投影到当前帧上可以获得图像点x′,同时可以估计它的深度信息zproj,如果一个动态物体对X进行了遮挡,则实际测量得到的深度信息z′会小于zproj,因此当某个点的′Δz=zproj-z′大于某个阈值τ时,将其归类为动态点。As a further improvement, the principle of the specific judgment in step c) of the present invention is as follows: the coordinates of the spatial point X can be projected onto the current frame to obtain the image point x′, and its depth information z proj can be estimated at the same time, if If a dynamic object blocks X, the actual measured depth information z′ will be smaller than z proj , so when a point’Δz=z proj -z′ is greater than a certain threshold τ, it is classified as dynamic point.
作为进一步地改进,本发明所述的τ值为0.5m。As a further improvement, the value of τ described in the present invention is 0.5m.
作为进一步地改进,本发明所述的步骤6)后,步骤7)前还包括如下步骤:送入建图线程,建图线程负责将相机位姿,彩色图,深度图生成点云图。As a further improvement, after step 6) described in the present invention, before step 7) also includes the following steps: send into the mapping thread, and the mapping thread is responsible for generating the point cloud image with the camera pose, color map, and depth map.
作为进一步地改进,本发明所述的RGBD相机为Kinect V2(微软第二代3D体感摄影机)。As a further improvement, the RGBD camera described in the present invention is Kinect V2 (Microsoft's second generation 3D somatosensory camera).
本发明的有益效果如下:The beneficial effects of the present invention are as follows:
本发明提出了一种语义分割相结合的三维重建方法,使用该方法有效的解决了三维重建在动态场景下相机位姿估计不准确以及动态物体重影的问题,利用SegNet网络对图像进行分割,利用先验知识排除属于动态区域的点,再利用多视图几何法进一步判断图像中属于动态部分的其他点,最后利用所有静态点进行相机位姿估计和三维场景重建。本发明对动态区域检测步骤进行了时间上的优化,可以实现实时重建的效果。在TUM的数据集上我们用相机轨迹绝对误差进行衡量,其均方根误差由先前的0.752提高至0.025。The present invention proposes a 3D reconstruction method combined with semantic segmentation, which effectively solves the problems of inaccurate camera pose estimation and ghosting of dynamic objects in 3D reconstruction in dynamic scenes, and uses the SegNet network to segment images, Use prior knowledge to exclude points belonging to the dynamic area, and then use the multi-view geometry method to further judge other points belonging to the dynamic part of the image, and finally use all static points for camera pose estimation and 3D scene reconstruction. The invention optimizes the time of the dynamic area detection step, and can realize the effect of real-time reconstruction. On the TUM dataset, we use the absolute error of the camera trajectory to measure, and its root mean square error has increased from the previous 0.752 to 0.025.
附图说明Description of drawings
图1是本发明系统的流程示意图;Fig. 1 is a schematic flow chart of the system of the present invention;
图2是多视图几何法判断动态点原理示意图。Fig. 2 is a schematic diagram of the principle of judging dynamic points by the multi-view geometry method.
具体实施方法Specific implementation method
本发明公开了一种基于RGBD相机的室内三维重建方法,,着重解决室内场景中的动态物体,装置主要由RGBD相机,带有GPU的PC机组成,RGBD相机为Kinect,图1是本发明系统的流程示意图;具体步骤如下:The present invention discloses an indoor three-dimensional reconstruction method based on RGBD camera, which focuses on solving dynamic objects in indoor scenes. The device is mainly composed of RGBD camera and PC with GPU. The RGBD camera is Kinect. Figure 1 is the system of the present invention Schematic diagram of the process; the specific steps are as follows:
标定Kinect相机,得到Kinect的彩色相机内参值和深度相机的内参值以及深度相机和彩色相机的转移矩阵。Calibrate the Kinect camera, get the Kinect color camera internal parameter value and depth camera internal parameter value and the transfer matrix of the depth camera and color camera.
采集场景图像:每一帧包括一张彩色图和深度图,利用Kinect的SDK将彩色图与深度图对齐。Collect scene images: each frame includes a color map and a depth map, and use the Kinect SDK to align the color map with the depth map.
提取特征点:利用ORB特征点算法提取出彩色图像中的特征点。Extract feature points: use the ORB feature point algorithm to extract feature points in the color image.
动态点剔除:本发明利用卷积神经网络和多视图几何法相结合的方式来进行动态点的剔除。具体方法如下:Dynamic point elimination: The present invention utilizes a combination of convolutional neural network and multi-view geometry method to eliminate dynamic points. The specific method is as follows:
1、利用重新训练的SegNet网络对彩色图像进行分割,得到每个像素的标签,得到的人和其他动物的区域属于动态区域,其内的特征点属于动态点,对这些动态点进行剔除操作。1. Use the retrained SegNet network to segment the color image to obtain the label of each pixel. The obtained areas of people and other animals belong to the dynamic area, and the feature points in it belong to the dynamic points, and these dynamic points are eliminated.
2、跟踪:将当前帧剔除后的特征点和上一帧或者上一个参考帧做特征点匹配,将利用速度模型或参考帧模型估计出当前帧的一个初始位姿2. Tracking: Match the feature points of the current frame with the feature points of the previous frame or the previous reference frame, and use the velocity model or reference frame model to estimate an initial pose of the current frame
3、多视图几何法判断动态点:通过比较某一空间特征点投影到当前帧上深度估计值与该点的深度测量值来进行判断,当其差值大于一定阈值是认为该点为动态点。3. Judgment of dynamic points by multi-view geometry method: Judgment is made by comparing the estimated depth value of a certain spatial feature point projected on the current frame with the measured depth value of the point. When the difference is greater than a certain threshold, the point is considered to be a dynamic point .
4、对多视图几何法得到动态点,对其在深度图上进行区域生长操作,得到动态区域。4. The dynamic point is obtained by the multi-view geometry method, and the region growing operation is performed on it on the depth map to obtain the dynamic region.
5、将卷积神经网络和多视图几何法得到的两个动态区域进行融合:融合的具体操作方法是对两个区域取并集。5. Fuse the two dynamic regions obtained by the convolutional neural network and the multi-view geometry method: the specific operation method of the fusion is to take the union of the two regions.
6、为了提高检测速度,每5帧进行一次动态区域检测。6. In order to improve the detection speed, a dynamic area detection is performed every 5 frames.
再跟踪:在剔除动态点后,重新利用速度模型和参考帧模型进行对当前帧位姿进行估计,在得到初始位姿后在局部地图中进行更精确的跟踪。Re-tracking: After removing dynamic points, re-use the velocity model and reference frame model to estimate the pose of the current frame, and perform more accurate tracking in the local map after obtaining the initial pose.
插入新关键帧:判断插入新关键帧的依据是:距离上一次插入关键帧超过了20帧以上,当前帧追踪到了到的地图点少于50个,当前帧与参考帧的重合比例少于90%。Insert a new key frame: The basis for judging the insertion of a new key frame is: the distance from the last inserted key frame exceeds 20 frames, the number of map points tracked by the current frame is less than 50, and the overlap ratio between the current frame and the reference frame is less than 90 %.
送入建图线程,建图线程负责将相机位姿,彩色图,深度图生成点云图。Send it to the mapping thread, which is responsible for generating a point cloud image from the camera pose, color map, and depth map.
局部地图优化:对于新加入的关键帧,找出其共视帧和这些共视帧可以观察到的地图点,进行BA优化。Local map optimization: For newly added key frames, find out their common view frames and map points that can be observed by these common view frames, and perform BA optimization.
回环检测:利用当前关键帧的词袋模型判断是否能与之前帧构成闭环,如果通过闭环一致性检,则计算闭环帧之间的相似变换,并对相邻帧进行闭环矫正。Loop detection: Use the bag-of-words model of the current key frame to judge whether it can form a closed loop with the previous frame. If it passes the closed-loop consistency check, calculate the similarity transformation between closed-loop frames, and perform closed-loop correction on adjacent frames.
下面通过具体实施例对本发明的技术方案作进一步地说明:The technical scheme of the present invention will be further described below by specific examples:
作为一种优选RGBD相机采用了KinectV2,具体步骤如下:As a preferred RGBD camera, KinectV2 is adopted, and the specific steps are as follows:
步骤一:使用棋盘格对KinectV2进行标定,通过标定算法得到得到Kinect的彩色相机内参值和深度相机的内参值以及深度相机和彩色相机的转移矩阵R,T;Step 1: Use the checkerboard to calibrate KinectV2, and get Kinect's internal parameter values of the color camera and depth camera and the transfer matrix R, T of the depth camera and color camera through the calibration algorithm;
步骤二:可以手持Kinect或者将Kinect置于移动机器人上,对室内场景进行采集,其中Kinect得到每一帧信息包括一张彩色图和深度图,需要利用Kinect的SDK将彩色图与深度图对齐。Step 2: You can hold the Kinect or put the Kinect on the mobile robot to collect the indoor scene. Kinect gets each frame information including a color map and a depth map. You need to use the Kinect SDK to align the color map with the depth map.
步骤三:对输入的每一帧图像进行跟踪,这一步包含了动态点部分的剔除以及对相机位姿的初始跟踪。Step 3: Track each frame of the input image. This step includes the removal of dynamic points and the initial tracking of the camera pose.
1、提取特征点:利用ORB特征点算法提取出彩色图像中的特征点。1. Extract feature points: Use the ORB feature point algorithm to extract feature points in the color image.
2、动态点检测与剔除:2. Dynamic point detection and elimination:
a)图像分割与识别:利用重新训练的SegNet网络对彩色图像进行分割,得到每个像素的标签,得到的人和其他动物的区域属于动态区域,其内的特征点属于动态点,对这些动态点进行剔除操作。a) Image segmentation and recognition: Use the retrained SegNet network to segment the color image to get the label of each pixel. The obtained areas of people and other animals belong to the dynamic area, and the feature points in it belong to the dynamic points. For these dynamic Click to delete.
b)跟踪:将当前帧剔除后的特征点和上一帧或者上一个参考帧做特征点匹配,将利用速度模型或参考帧模型估计出当前帧的一个初始估计位姿b) Tracking: match the feature points of the current frame with the feature points of the previous frame or the previous reference frame, and use the velocity model or the reference frame model to estimate an initial estimated pose of the current frame
c)多视图几何法判断动态点:图2是多视图几何法判断动态点原理示意图;利用上一步求得的初始估计位姿对特征点进行判断是否属于动态点,具体判断的原理如下:将空间点X的坐标投影到当前帧上可以获得图像点x′,同时可以估计它的深度信息zproj,如果一个动态物体对X进行了遮挡,则实际测量得到的深度信息z′会小于zproj,如图2所示。因此当某个点的′Δz=zproj-z′大于某个阈值τ时,将其归类为动态点。经过多次实验我们将τ设置为0.5m。c) Multi-view geometry method to judge dynamic points: Figure 2 is a schematic diagram of the principle of multi-view geometry method to judge dynamic points; use the initial estimated pose obtained in the previous step to judge whether a feature point belongs to a dynamic point, and the specific judgment principle is as follows: The coordinates of the spatial point X are projected onto the current frame to obtain the image point x′, and its depth information z proj can be estimated at the same time. If a dynamic object blocks X, the actual measured depth information z′ will be smaller than z proj ,as shown in picture 2. Therefore, when 'Δz=z proj -z' of a certain point is greater than a certain threshold τ, it is classified as a dynamic point. After many experiments we set τ as 0.5m.
d)对多视图几何法得到动态点,对其在深度图上进行区域生长操作,得到动态区域。d) The multi-view geometry method is used to obtain dynamic points, and the region growing operation is performed on the depth map to obtain dynamic regions.
e)将卷积神经网络和多视图几何法得到的两个动态区域进行融合:融合的具体操作方法是对两个区域取并集。e) Fuse the two dynamic regions obtained by the convolutional neural network and the multi-view geometry method: the specific operation method of fusion is to take the union of the two regions.
f)为了提高检测速度,每5帧进行一次动态区域检测,对于未进行动态区域检测的帧,需要同已经进行动态区域检测的帧进行特征点匹配。利用匹配后落在动态区域外的特征点进行相机位姿估计。f) In order to improve the detection speed, a dynamic area detection is performed every 5 frames. For frames without dynamic area detection, it is necessary to perform feature point matching with frames that have undergone dynamic area detection. The camera pose estimation is performed using the matched feature points that fall outside the dynamic region.
3、再跟踪:在利用多视图几何法进一步剔除动态点后,重新利用速度模型和参考帧模型进行对当前帧位姿进行估计,在得到初始位姿后在局部地图中进行更精确的跟踪。3. Re-tracking: After using the multi-view geometry method to further eliminate dynamic points, re-use the velocity model and the reference frame model to estimate the pose of the current frame, and perform more accurate tracking in the local map after obtaining the initial pose.
4、插入新关键帧:判断插入新关键帧的依据是:距离上一次插入关键帧超过了20帧以上,当前帧追踪到了到的地图点少于50个,当前帧与参考帧的重合比例少于90%。4. Insert a new key frame: The basis for judging the insertion of a new key frame is: the distance from the last inserted key frame exceeds 20 frames, the number of map points tracked by the current frame is less than 50, and the overlap ratio between the current frame and the reference frame is small at 90%.
步骤四:局部地图优化:对新加入的关键帧、新加入关键帧的所有共视帧及所有共视帧中的地图点进行BA优化。Step 4: Local map optimization: BA optimization is performed on the newly added key frame, all common view frames of the newly added key frame, and map points in all common view frames.
如果设观测方程为z=h(ξ,p),这里ξ是相机位姿的李代数表示,p表示路标点的世界坐标,而观测数据z表示像素坐标z=[us,vs]T,因此投影误差表示为e=z-h(ξ,p),对于BA优化是要将共视帧和共视帧中的地图点考虑进来,因此总的误差项写为If the observation equation is set as z=h(ξ, p), where ξ is the Lie algebra representation of the camera pose, p represents the world coordinates of landmark points, and the observation data z represents the pixel coordinates z=[u s , v s ] T , so the projection error is expressed as e=zh(ξ, p), for BA optimization is to take the common-view frame and the map points in the common-view frame into account, so the total error term is written as
也就是让上述误差项最小,其中eij表示在第i个位姿观测第j个特征点的误差。That is to minimize the above error term, where e ij represents the error of observing the j-th feature point at the i-th pose.
步骤五:回环检测,利用当前关键帧的词袋模型判断是否与之前帧构成闭环,如果通过闭环一致性检,则计算闭环帧之间的相似变换,相邻帧进行闭环矫正。Step 5: Loop closure detection. Use the bag-of-words model of the current key frame to determine whether it forms a closed loop with the previous frame. If it passes the closed-loop consistency check, calculate the similarity transformation between closed-loop frames, and perform closed-loop correction on adjacent frames.
本发明最终的重建结果是:可以完整的还原出室内静态场景,并输出室内场景的彩色点云地图,点云地图中有效地消除了人等动态物体产生的残影。The final reconstruction result of the present invention is: the indoor static scene can be fully restored, and the colored point cloud map of the indoor scene can be output, and the afterimages produced by dynamic objects such as people are effectively eliminated in the point cloud map.
以上所述并非是对本发明的限制,应当指出,对于本技术领域的普通技术人员来说,在不脱离本发明实质范围的前提下,还可以做出若干变化、改型、添加或替换,这些改进和润饰也应视为本发明的保护范围。The above is not a limitation of the present invention. It should be pointed out that for those of ordinary skill in the art, some changes, modifications, additions or substitutions can be made without departing from the essential scope of the present invention. These Improvements and retouches should also be considered within the protection scope of the present invention.
Claims (6)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910572096.1A CN110349250B (en) | 2019-06-28 | 2019-06-28 | RGBD camera-based three-dimensional reconstruction method for indoor dynamic scene |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910572096.1A CN110349250B (en) | 2019-06-28 | 2019-06-28 | RGBD camera-based three-dimensional reconstruction method for indoor dynamic scene |
Publications (2)
Publication Number | Publication Date |
---|---|
CN110349250A true CN110349250A (en) | 2019-10-18 |
CN110349250B CN110349250B (en) | 2020-12-22 |
Family
ID=68177197
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201910572096.1A Active CN110349250B (en) | 2019-06-28 | 2019-06-28 | RGBD camera-based three-dimensional reconstruction method for indoor dynamic scene |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN110349250B (en) |
Cited By (31)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111161318A (en) * | 2019-12-30 | 2020-05-15 | 广东工业大学 | Dynamic scene SLAM method based on YOLO algorithm and GMS feature matching |
CN111160107A (en) * | 2019-12-05 | 2020-05-15 | 东南大学 | A dynamic region detection method based on feature matching |
CN111402336A (en) * | 2020-03-23 | 2020-07-10 | 中国科学院自动化研究所 | Semantic S L AM-based dynamic environment camera pose estimation and semantic map construction method |
CN111390975A (en) * | 2020-04-27 | 2020-07-10 | 浙江库科自动化科技有限公司 | Inspection intelligent robot with air pipe removing function and inspection method thereof |
CN111462179A (en) * | 2020-03-26 | 2020-07-28 | 北京百度网讯科技有限公司 | Three-dimensional object tracking method and device and electronic equipment |
CN111709982A (en) * | 2020-05-22 | 2020-09-25 | 浙江四点灵机器人股份有限公司 | Three-dimensional reconstruction method for dynamic environment |
CN111724439A (en) * | 2019-11-29 | 2020-09-29 | 中国科学院上海微系统与信息技术研究所 | A visual positioning method and device in a dynamic scene |
CN111739037A (en) * | 2020-07-31 | 2020-10-02 | 之江实验室 | A Semantic Segmentation Method for RGB-D Images of Indoor Scenes |
CN111914832A (en) * | 2020-06-03 | 2020-11-10 | 华南理工大学 | A SLAM method for RGB-D cameras in dynamic scenes |
CN112037268A (en) * | 2020-09-02 | 2020-12-04 | 中国科学技术大学 | Environment sensing method based on probability transfer model in dynamic scene |
CN112037261A (en) * | 2020-09-03 | 2020-12-04 | 北京华捷艾米科技有限公司 | A kind of image dynamic feature removal method and device |
CN112101160A (en) * | 2020-09-04 | 2020-12-18 | 浙江大学 | A Binocular Semantic SLAM Method for Autonomous Driving Scenarios |
CN112220444A (en) * | 2019-11-20 | 2021-01-15 | 北京健康有益科技有限公司 | Pupil distance measuring method and device based on depth camera |
CN112258618A (en) * | 2020-11-04 | 2021-01-22 | 中国科学院空天信息创新研究院 | Semantic mapping and localization method based on fusion of prior laser point cloud and depth map |
CN112435262A (en) * | 2020-11-27 | 2021-03-02 | 广东电网有限责任公司肇庆供电局 | Dynamic environment information detection method based on semantic segmentation network and multi-view geometry |
CN112530014A (en) * | 2020-12-18 | 2021-03-19 | 北京理工大学重庆创新中心 | Multi-unmanned aerial vehicle indoor scene three-dimensional reconstruction method and device |
CN112651357A (en) * | 2020-12-30 | 2021-04-13 | 浙江商汤科技开发有限公司 | Segmentation method of target object in image, three-dimensional reconstruction method and related device |
CN112767409A (en) * | 2019-11-05 | 2021-05-07 | 珠海格力电器股份有限公司 | Image processing method and device before positioning, storage medium and computer equipment |
CN112802053A (en) * | 2021-01-27 | 2021-05-14 | 广东工业大学 | Dynamic object detection method for dense mapping in dynamic environment |
CN112802186A (en) * | 2021-01-27 | 2021-05-14 | 清华大学 | Dynamic scene real-time three-dimensional reconstruction method based on binarization characteristic coding matching |
CN112907677A (en) * | 2019-12-04 | 2021-06-04 | 杭州海康威视数字技术股份有限公司 | Camera calibration method and device for single-frame image and storage medium |
CN113256663A (en) * | 2021-05-13 | 2021-08-13 | 深圳亿嘉和科技研发有限公司 | Dynamic feature eliminating method and angular point detection algorithm thereof |
CN113447014A (en) * | 2021-08-30 | 2021-09-28 | 深圳市大道智创科技有限公司 | Indoor mobile robot, mapping method, positioning method, and mapping positioning device |
CN113673524A (en) * | 2021-07-05 | 2021-11-19 | 北京物资学院 | Method and device for removing dynamic characteristic points of warehouse semi-structured environment |
CN114565656A (en) * | 2022-02-10 | 2022-05-31 | 北京箩筐时空数据技术有限公司 | Camera pose prediction method and device, storage medium and computer equipment |
WO2022110514A1 (en) * | 2020-11-27 | 2022-06-02 | 叠境数字科技(上海)有限公司 | Image interpolation method and apparatus employing rgb-d image and multi-camera system |
CN114723672A (en) * | 2022-03-09 | 2022-07-08 | 杭州易现先进科技有限公司 | Method, system, device and medium for three-dimensional reconstruction data acquisition and verification |
CN114972656A (en) * | 2022-06-23 | 2022-08-30 | 安徽工业大学 | Dynamic scene vision SLAM optimization method based on semantic segmentation network |
CN116206068A (en) * | 2023-04-28 | 2023-06-02 | 北京科技大学 | Three-dimensional driving scene generation and construction method and device based on real data set |
CN116258817A (en) * | 2023-02-16 | 2023-06-13 | 浙江大学 | A method and system for constructing an autonomous driving digital twin scene based on multi-view 3D reconstruction |
CN116452776A (en) * | 2023-06-19 | 2023-07-18 | 国网浙江省电力有限公司湖州供电公司 | Low-carbon substation scene reconstruction method based on vision synchronous positioning and mapping system |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103049929A (en) * | 2012-11-20 | 2013-04-17 | 浙江大学 | Multi-camera dynamic scene 3D (three-dimensional) rebuilding method based on joint optimization |
US20130182894A1 (en) * | 2012-01-18 | 2013-07-18 | Samsung Electronics Co., Ltd. | Method and apparatus for camera tracking |
CN107193279A (en) * | 2017-05-09 | 2017-09-22 | 复旦大学 | Robot localization and map structuring system based on monocular vision and IMU information |
CN107833236A (en) * | 2017-10-31 | 2018-03-23 | 中国科学院电子学研究所 | Semantic vision positioning system and method are combined under a kind of dynamic environment |
CN108596974A (en) * | 2018-04-04 | 2018-09-28 | 清华大学 | Dynamic scene robot localization builds drawing system and method |
KR20180112622A (en) * | 2017-04-04 | 2018-10-12 | 엘지전자 주식회사 | Method of configuring position based on identification of fixed object and moving object and robot implementing thereof |
CN109387204A (en) * | 2018-09-26 | 2019-02-26 | 东北大学 | The synchronous positioning of the mobile robot of dynamic environment and patterning process in faced chamber |
-
2019
- 2019-06-28 CN CN201910572096.1A patent/CN110349250B/en active Active
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130182894A1 (en) * | 2012-01-18 | 2013-07-18 | Samsung Electronics Co., Ltd. | Method and apparatus for camera tracking |
CN103049929A (en) * | 2012-11-20 | 2013-04-17 | 浙江大学 | Multi-camera dynamic scene 3D (three-dimensional) rebuilding method based on joint optimization |
KR20180112622A (en) * | 2017-04-04 | 2018-10-12 | 엘지전자 주식회사 | Method of configuring position based on identification of fixed object and moving object and robot implementing thereof |
CN107193279A (en) * | 2017-05-09 | 2017-09-22 | 复旦大学 | Robot localization and map structuring system based on monocular vision and IMU information |
CN107833236A (en) * | 2017-10-31 | 2018-03-23 | 中国科学院电子学研究所 | Semantic vision positioning system and method are combined under a kind of dynamic environment |
CN108596974A (en) * | 2018-04-04 | 2018-09-28 | 清华大学 | Dynamic scene robot localization builds drawing system and method |
CN109387204A (en) * | 2018-09-26 | 2019-02-26 | 东北大学 | The synchronous positioning of the mobile robot of dynamic environment and patterning process in faced chamber |
Non-Patent Citations (2)
Title |
---|
JIANBO ZHANG,ET AL: "《Semantic Segmentation based Dense RGB-D SLAM in Dynamic Environments》", 《2019 3RD INTERNATIONAL CONFERENCE ON ARTIFICIAL INTELLIGENCE, AUTOMATION AND CONTROL TECHNOLOGIES (AIACT 2019) 》 * |
代具亭: "《基于RGB-D视频序列的大尺度场景三维语义表面重建技术研究》", 《中国博士学位论文全文数据库 信息科技辑》 * |
Cited By (48)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112767409A (en) * | 2019-11-05 | 2021-05-07 | 珠海格力电器股份有限公司 | Image processing method and device before positioning, storage medium and computer equipment |
CN112220444A (en) * | 2019-11-20 | 2021-01-15 | 北京健康有益科技有限公司 | Pupil distance measuring method and device based on depth camera |
CN112220444B (en) * | 2019-11-20 | 2021-06-29 | 北京健康有益科技有限公司 | Pupil distance measuring method and device based on depth camera |
CN111724439A (en) * | 2019-11-29 | 2020-09-29 | 中国科学院上海微系统与信息技术研究所 | A visual positioning method and device in a dynamic scene |
CN111724439B (en) * | 2019-11-29 | 2024-05-17 | 中国科学院上海微系统与信息技术研究所 | Visual positioning method and device under dynamic scene |
CN112907677A (en) * | 2019-12-04 | 2021-06-04 | 杭州海康威视数字技术股份有限公司 | Camera calibration method and device for single-frame image and storage medium |
CN112907677B (en) * | 2019-12-04 | 2023-07-25 | 杭州海康威视数字技术股份有限公司 | Camera calibration method and device for single-frame image and storage medium |
CN111160107A (en) * | 2019-12-05 | 2020-05-15 | 东南大学 | A dynamic region detection method based on feature matching |
CN111161318A (en) * | 2019-12-30 | 2020-05-15 | 广东工业大学 | Dynamic scene SLAM method based on YOLO algorithm and GMS feature matching |
CN111402336B (en) * | 2020-03-23 | 2024-03-12 | 中国科学院自动化研究所 | Semantic SLAM-based dynamic environment camera pose estimation and semantic map construction method |
CN111402336A (en) * | 2020-03-23 | 2020-07-10 | 中国科学院自动化研究所 | Semantic S L AM-based dynamic environment camera pose estimation and semantic map construction method |
CN111462179B (en) * | 2020-03-26 | 2023-06-27 | 北京百度网讯科技有限公司 | Three-dimensional object tracking method and device and electronic equipment |
CN111462179A (en) * | 2020-03-26 | 2020-07-28 | 北京百度网讯科技有限公司 | Three-dimensional object tracking method and device and electronic equipment |
CN111390975A (en) * | 2020-04-27 | 2020-07-10 | 浙江库科自动化科技有限公司 | Inspection intelligent robot with air pipe removing function and inspection method thereof |
CN111709982A (en) * | 2020-05-22 | 2020-09-25 | 浙江四点灵机器人股份有限公司 | Three-dimensional reconstruction method for dynamic environment |
CN111914832A (en) * | 2020-06-03 | 2020-11-10 | 华南理工大学 | A SLAM method for RGB-D cameras in dynamic scenes |
CN111914832B (en) * | 2020-06-03 | 2023-06-13 | 华南理工大学 | SLAM method of RGB-D camera under dynamic scene |
CN111739037B (en) * | 2020-07-31 | 2020-12-01 | 之江实验室 | A Semantic Segmentation Method for RGB-D Images of Indoor Scenes |
CN111739037A (en) * | 2020-07-31 | 2020-10-02 | 之江实验室 | A Semantic Segmentation Method for RGB-D Images of Indoor Scenes |
CN112037268A (en) * | 2020-09-02 | 2020-12-04 | 中国科学技术大学 | Environment sensing method based on probability transfer model in dynamic scene |
CN112037268B (en) * | 2020-09-02 | 2022-09-02 | 中国科学技术大学 | Environment sensing method based on probability transfer model in dynamic scene |
CN112037261A (en) * | 2020-09-03 | 2020-12-04 | 北京华捷艾米科技有限公司 | A kind of image dynamic feature removal method and device |
CN112101160A (en) * | 2020-09-04 | 2020-12-18 | 浙江大学 | A Binocular Semantic SLAM Method for Autonomous Driving Scenarios |
CN112101160B (en) * | 2020-09-04 | 2024-01-05 | 浙江大学 | Binocular semantic SLAM method for automatic driving scene |
CN112258618B (en) * | 2020-11-04 | 2021-05-14 | 中国科学院空天信息创新研究院 | Semantic mapping and positioning method based on fusion of prior laser point cloud and depth map |
CN112258618A (en) * | 2020-11-04 | 2021-01-22 | 中国科学院空天信息创新研究院 | Semantic mapping and localization method based on fusion of prior laser point cloud and depth map |
WO2022110514A1 (en) * | 2020-11-27 | 2022-06-02 | 叠境数字科技(上海)有限公司 | Image interpolation method and apparatus employing rgb-d image and multi-camera system |
CN112435262A (en) * | 2020-11-27 | 2021-03-02 | 广东电网有限责任公司肇庆供电局 | Dynamic environment information detection method based on semantic segmentation network and multi-view geometry |
CN112530014A (en) * | 2020-12-18 | 2021-03-19 | 北京理工大学重庆创新中心 | Multi-unmanned aerial vehicle indoor scene three-dimensional reconstruction method and device |
CN112530014B (en) * | 2020-12-18 | 2023-07-25 | 北京理工大学重庆创新中心 | Three-dimensional reconstruction method and device for indoor scene of multiple unmanned aerial vehicles |
CN112651357B (en) * | 2020-12-30 | 2024-05-24 | 浙江商汤科技开发有限公司 | Method for segmenting target object in image, three-dimensional reconstruction method and related device |
CN112651357A (en) * | 2020-12-30 | 2021-04-13 | 浙江商汤科技开发有限公司 | Segmentation method of target object in image, three-dimensional reconstruction method and related device |
CN112802186A (en) * | 2021-01-27 | 2021-05-14 | 清华大学 | Dynamic scene real-time three-dimensional reconstruction method based on binarization characteristic coding matching |
CN112802186B (en) * | 2021-01-27 | 2022-06-24 | 清华大学 | Real-time 3D reconstruction method of dynamic scene based on binary feature code matching |
CN112802053A (en) * | 2021-01-27 | 2021-05-14 | 广东工业大学 | Dynamic object detection method for dense mapping in dynamic environment |
CN113256663A (en) * | 2021-05-13 | 2021-08-13 | 深圳亿嘉和科技研发有限公司 | Dynamic feature eliminating method and angular point detection algorithm thereof |
CN113673524A (en) * | 2021-07-05 | 2021-11-19 | 北京物资学院 | Method and device for removing dynamic characteristic points of warehouse semi-structured environment |
CN113447014A (en) * | 2021-08-30 | 2021-09-28 | 深圳市大道智创科技有限公司 | Indoor mobile robot, mapping method, positioning method, and mapping positioning device |
CN114565656A (en) * | 2022-02-10 | 2022-05-31 | 北京箩筐时空数据技术有限公司 | Camera pose prediction method and device, storage medium and computer equipment |
CN114723672A (en) * | 2022-03-09 | 2022-07-08 | 杭州易现先进科技有限公司 | Method, system, device and medium for three-dimensional reconstruction data acquisition and verification |
CN114723672B (en) * | 2022-03-09 | 2024-08-20 | 杭州易现先进科技有限公司 | Method, system, device and medium for three-dimensional reconstruction data acquisition and verification |
CN114972656B (en) * | 2022-06-23 | 2024-08-16 | 安徽工业大学 | A dynamic scene visual SLAM optimization method based on semantic segmentation network |
CN114972656A (en) * | 2022-06-23 | 2022-08-30 | 安徽工业大学 | Dynamic scene vision SLAM optimization method based on semantic segmentation network |
CN116258817B (en) * | 2023-02-16 | 2024-01-30 | 浙江大学 | A method and system for constructing autonomous driving digital twin scenes based on multi-view three-dimensional reconstruction |
CN116258817A (en) * | 2023-02-16 | 2023-06-13 | 浙江大学 | A method and system for constructing an autonomous driving digital twin scene based on multi-view 3D reconstruction |
CN116206068A (en) * | 2023-04-28 | 2023-06-02 | 北京科技大学 | Three-dimensional driving scene generation and construction method and device based on real data set |
CN116452776A (en) * | 2023-06-19 | 2023-07-18 | 国网浙江省电力有限公司湖州供电公司 | Low-carbon substation scene reconstruction method based on vision synchronous positioning and mapping system |
CN116452776B (en) * | 2023-06-19 | 2023-10-20 | 国网浙江省电力有限公司湖州供电公司 | Low-carbon substation scene reconstruction method based on visual simultaneous positioning and mapping system |
Also Published As
Publication number | Publication date |
---|---|
CN110349250B (en) | 2020-12-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN110349250B (en) | RGBD camera-based three-dimensional reconstruction method for indoor dynamic scene | |
CN114384920B (en) | Dynamic obstacle avoidance method based on real-time construction of local grid map | |
CN111583369B (en) | Laser SLAM method based on facial line angular point feature extraction | |
CN113345018B (en) | Laser monocular vision fusion positioning mapping method in dynamic scene | |
CN109597087B (en) | A 3D object detection method based on point cloud data | |
CN114782626B (en) | Substation scene mapping and positioning optimization method based on laser and vision fusion | |
Xiao et al. | Monocular vehicle self-localization method based on compact semantic map | |
CN112734765A (en) | Mobile robot positioning method, system and medium based on example segmentation and multi-sensor fusion | |
Murali et al. | Utilizing semantic visual landmarks for precise vehicle navigation | |
Sujiwo et al. | Monocular vision-based localization using ORB-SLAM with LIDAR-aided mapping in real-world robot challenge | |
CN108648274A (en) | A kind of cognition point cloud map creation system of vision SLAM | |
CN112734841A (en) | Method for realizing positioning by using wheel type odometer-IMU and monocular camera | |
CN114964236A (en) | Mapping and vehicle positioning system and method for underground parking lot environment | |
CN111998862A (en) | Dense binocular SLAM method based on BNN | |
Hayakawa et al. | Ego-motion and surrounding vehicle state estimation using a monocular camera | |
CN115830070A (en) | Infrared laser fusion positioning method for inspection robot of traction substation | |
CN118603077A (en) | A quadruped robot inspection map construction system and method based on multi-sensor fusion | |
Tian et al. | Vision-based mapping of lane semantics and topology for intelligent vehicles | |
CN119440090A (en) | Autonomous obstacle avoidance method, system, device and storage medium for inspection drone | |
CN113720323B (en) | Monocular vision inertial navigation SLAM method and device based on point-line feature fusion | |
Sujiwo et al. | Localization based on multiple visual-metric maps | |
CN111696147B (en) | Depth estimation method based on improved YOLOv3 model | |
CN117635651A (en) | A dynamic environment SLAM method based on YOLOv8 instance segmentation | |
CN116630561A (en) | Outdoor visual positioning and mapping method based on quadric surface initialization of robustness | |
CN115482282A (en) | Dynamic SLAM method with multi-target tracking capability in autonomous driving scenarios |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |