CN110349250A - A kind of three-dimensional rebuilding method of the indoor dynamic scene based on RGBD camera - Google Patents

A kind of three-dimensional rebuilding method of the indoor dynamic scene based on RGBD camera Download PDF

Info

Publication number
CN110349250A
CN110349250A CN201910572096.1A CN201910572096A CN110349250A CN 110349250 A CN110349250 A CN 110349250A CN 201910572096 A CN201910572096 A CN 201910572096A CN 110349250 A CN110349250 A CN 110349250A
Authority
CN
China
Prior art keywords
point
dynamic
frame
camera
convolutional neural
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201910572096.1A
Other languages
Chinese (zh)
Other versions
CN110349250B (en
Inventor
林斌
曹权
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhejiang University ZJU
Original Assignee
Zhejiang University ZJU
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhejiang University ZJU filed Critical Zhejiang University ZJU
Priority to CN201910572096.1A priority Critical patent/CN110349250B/en
Publication of CN110349250A publication Critical patent/CN110349250A/en
Application granted granted Critical
Publication of CN110349250B publication Critical patent/CN110349250B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Mathematical Physics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Biophysics (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Biomedical Technology (AREA)
  • Computational Linguistics (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Quality & Reliability (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a kind of three-dimensional rebuilding methods of indoor dynamic scene based on RGBD camera, including demarcating RGBD camera, acquire scene image, extract characteristic point, dynamic point detection and rejecting, by convolutional neural networks and multi-view geometry method combine in the way of carry out the rejecting of dynamic point, it tracks again, it is inserted into new key frame, local map optimization, winding detection and etc., the problem of camera pose estimates inaccuracy and dynamic object ghost image under dynamic scene using this method effective solution three-dimensional reconstruction, the present invention has carried out temporal optimization to dynamic area detecting step, the effect of real-time reconstruction may be implemented.We are measured with camera track absolute error on the data set of TUM, and root-mean-square error is improved by previous 0.752 to 0.025.

Description

A kind of three-dimensional rebuilding method of the indoor dynamic scene based on RGBD camera
Technical field
The invention belongs to three-dimensional imaging fields, are the three of a kind of indoor dynamic scene based on RGBD camera specifically Tie up method for reconstructing.
Background technique
The progress of computer technology in recent years, AR, VR technology are increasingly becoming one of hot fields of research, in intelligence Household field has important application in virtual shopping field, and how effectively to rebuild the scene of surrounding is research therein One of direction.With the progress of artificial intelligence technology, in automatic driving field, unmanned plane autonomous flight field is also needed Solve the building and the orientation problem of itself of surrounding scene.
Above problem of the appearance very good solution of SLAM (simultaneous localization and mapping) technology.SLAM system utilizes Sensor on object (such as monocular, binocular camera in vision SLAM) is equipped on to obtain external information, utilizes these information Itself position and posture are estimated, construct map as needed.
At the same time, the neural network of deep learning makes us being pleasantly surprised in the performance of image recognition and segmentation field in recent years, The networks such as Alex-Net, Google-Net, VGG, Res-Net constantly refresh the accuracy rate of target identification data collection.Therefore with The semantic SLAM system that deep learning combines has new development.Semantic SLAM system to around scene rebuild it is same When the object in scene can be identified, applied and the reason of robot scene to around can be improved in robot field Solution and cognition can complete more complicated task for robot and provide possibility.Being applied can be real in automatic Pilot field Existing automatic obstacle avoidance, the functions such as danger early warning, therefore semanteme SLAM has broad application prospects.
Positioning under traditional SLAM method very good solution static scene and figure problem is built, but is having dynamic object Scene in, the positioning of these methods and the accuracy for building figure are very poor, because traditional SLAM method is difficult to differentiate between out goer Body and stationary body, and they carry out identical processing.But in fact, the spy of characteristic point and static background that dynamic object extracts Inconsistent when sign point movement, this can seriously affect the positioning of camera position, so that the result of figure is built in influence.In practical application In, it is also necessary for eliminating dynamic object, such as in the path planning and navigation of sweeping robot, if cannot eliminate People, dog, the dynamic objects such as cat can make the guidance path of robot generate deviation.
The present invention solves in SLAM system the track estimation inaccuracy under dynamic scene, and when map reconstruction generates ghost image Problem.The present invention joined convolutional neural networks in SLAM system, is split using convolutional neural networks to object, utilize Priori knowledge and multi-view geometry method judge dynamic point, to eliminate influence of the dynamic point to scene.
Summary of the invention
The purpose of the present invention is existing three-dimensional rebuilding method under dynamic scene there are aiming at the problem that propose a kind of base It is to be carried out using depth camera and color camera to indoor scene in the three-dimensional rebuilding method of RGBD camera and convolutional neural networks Three-dimensional reconstruction eliminates dynamic object to the final influence for rebuilding effect using the method for semantic segmentation.
The present invention is achieved through the following technical solutions:
The invention discloses a kind of three-dimensional rebuilding method based on RGBD camera and convolutional neural networks, three-dimensional rebuilding method Device used mainly includes RGBD camera, the PC machine with GPU, the specific steps are as follows:
1) it, demarcates RGBD camera: obtaining the color camera internal reference value of RGBD camera and the internal reference value and depth of depth camera Spend the transfer matrix of camera and color camera；
2), acquire scene image: each frame includes a cromogram and depth map, will be colored using the SDK of RGBD camera Figure is aligned with depth map；
3) it, extracts characteristic point: extracting the characteristic point in color image using ORB characteristic point algorithm；
4), dynamic point detection is with rejecting: carrying out in the way of being combined by convolutional neural networks and multi-view geometry method The rejecting of dynamic point；
5) it, tracks again: after further rejecting dynamic point using multi-view geometry method, re-using rate pattern and reference Frame model estimate present frame pose, is more accurately tracked in local map after obtaining initial pose；
6), be inserted into new key frame: the foundation that new key frame is inserted into judgement is that the last insertion key frame of distance has been more than 20 It is more than frame, present frame tracked to point map less than 50, the coincidence ratio of present frame and reference frame is less than 90%；
7), local map optimizes: to the key frame being newly added, new all of key frame are added all regard depending on frame and altogether frame altogether In point map carry out BA optimization；
8), winding detects: judging whether that previous frame constitutes closed loop therewith using the bag of words of current key frame, if crossing closed loop Consistency inspection, then calculate the similarity transformation between closed loop frame, and consecutive frame carries out closed loop correction.
As a further improvement, dynamic point detection includes the following steps: with rejecting in step 4) of the present invention
A), image segmentation and identification: a kind of SegNet network (semanteme based on convolutional neural networks of re -training is utilized Segmentation network) color image is split, the label of each pixel is obtained, the region of obtained people and other animals belongs to dynamic State region, in characteristic point belong to dynamic point, rejecting operation is carried out to these dynamic points；
B), track: characteristic point and previous frame or a upper reference frame after present frame is rejected do Feature Points Matching, will An initial pose of present frame is estimated using rate pattern or reference frame model；
C), multi-view geometry method judges dynamic point: projecting to depth on present frame by comparing a certain space characteristics point and estimates The depth measurement of evaluation and the point is judged, when its difference is greater than certain threshold value is to think that the point is dynamic point；
D), dynamic point is obtained to multi-view geometry method, region growing operation is carried out on depth map to it, obtains dynamic area Domain；
E), two dynamic areas for obtaining convolutional neural networks and multi-view geometry method are merged, fusion it is specific Operating method is to take union to two regions.
F), as a further improvement, in order to improve detection speed, every 5 frame carries out a dynamic area and detects, i.e., and every 5 Frame repeats above a)-e) step.
As a further improvement, the principle specifically judged in step c) of the present invention is as follows: by spatial point X's Picture point x ' can be obtained on coordinate projection to present frame, while can estimate its depth information zprojIf a dynamic Object blocks X, then the depth information z ' that actual measurement obtains can be less than zproj, therefore when some point ' Δ z= zprojWhen-z ' is greater than some threshold tau, it is classified as dynamic point.
As a further improvement, τ value of the present invention is 0.5m.
As a further improvement, further including following steps before step 7) after step 6) of the present invention: feeding builds figure Thread is built figure line journey and is responsible for camera pose, cromogram, and depth map generates point cloud chart.
As a further improvement, RGBD camera of the present invention is Kinect V2 (Microsoft's second generation 3D body-sensing photography Machine).
Beneficial effects of the present invention are as follows:
The invention proposes a kind of three-dimensional rebuilding methods that semantic segmentation combines, and use this method effective solution three Dimension is reconstituted in the problem of inaccurate camera pose estimation under dynamic scene and dynamic object ghost image, using SegNet network to figure As being split, the point for belonging to dynamic area is excluded using priori knowledge, multi-view geometry method is recycled further to judge image In belong to other points of dynamic part, finally all static points is utilized to carry out the estimation of camera pose and 3 D scene rebuilding.This hair It is bright that temporal optimization has been carried out to dynamic area detecting step, the effect of real-time reconstruction may be implemented.On the data set of TUM We are measured with camera track absolute error, and root-mean-square error is improved by previous 0.752 to 0.025.
Detailed description of the invention
Fig. 1 is the flow diagram of present system；
Fig. 2 is that multi-view geometry method judges dynamic point schematic illustration.
Specific implementation method
The invention discloses a kind of indoor three-dimensional rebuilding method based on RGBD camera, it focuses on solving in indoor scene Dynamic object, device is mainly by RGBD camera, and the PC machine composition with GPU, RGBD camera is Kinect, and Fig. 1 is system of the present invention The flow diagram of system；Specific step is as follows:
Kinect camera is demarcated, the color camera internal reference value of Kinect and the internal reference value of depth camera and depth phase are obtained The transfer matrix of machine and color camera.
Acquire scene image: each frame includes a cromogram and depth map, using the SDK of Kinect by cromogram and depth Degree figure alignment.
It extracts characteristic point: extracting the characteristic point in color image using ORB characteristic point algorithm.
Dynamic point reject: the present invention by convolutional neural networks and multi-view geometry method combine in the way of come carry out dynamic The rejecting of point.The specific method is as follows:
1, color image is split using the SegNet network of re -training, obtains the label of each pixel, obtains People and the regions of other animals belong to dynamic area, in characteristic point belong to dynamic point, these dynamic points are rejected Operation.
2, track: characteristic point and previous frame or a upper reference frame after present frame is rejected do Feature Points Matching, will An initial pose of present frame is estimated using rate pattern or reference frame model
3, multi-view geometry method judges dynamic point: projecting to estimation of Depth on present frame by comparing a certain space characteristics point The depth measurement of value and the point is judged, when its difference is greater than certain threshold value is to think that the point is dynamic point.
4, dynamic point is obtained to multi-view geometry method, region growing operation is carried out on depth map to it, obtains dynamic area Domain.
5, two dynamic areas for obtaining convolutional neural networks and multi-view geometry method are merged: the specific behaviour of fusion It is that union is taken to two regions as method.
6, in order to improve detection speed, every 5 frame carries out a dynamic area detection.
Track again: after rejecting dynamic point, re-use rate pattern and reference frame model carry out to present frame pose into Row estimation, is more accurately tracked in local map after obtaining initial pose.
Be inserted into new key frame: the foundation that new key frame is inserted into judgement is: the last insertion key frame of distance has been more than 20 frames More than, present frame tracked to point map less than 50, the coincidence ratio of present frame and reference frame is less than 90%.
Feeding builds figure line journey, builds figure line journey and is responsible for camera pose, cromogram, depth map generates point cloud chart.
Local map optimization: it for the key frame being newly added, finds out it and regards frame altogether and be observed that altogether depending on frame with these Point map carries out BA optimization.
Winding detection: judge whether previous frame to constitute closed loop therewith using the bag of words of current key frame, if passed through The inspection of closed loop consistency then calculates the similarity transformation between closed loop frame, and carries out closed loop correction to consecutive frame.
Technical solution of the present invention is further described below by specific embodiment:
KinectV2 is used as a kind of preferred RGBD camera, the specific steps are as follows:
Step 1: demarcating KinectV2 using gridiron pattern, obtains the colour of Kinect by calibration algorithm The transfer matrix R, T of the internal reference value and depth camera and color camera of camera internal reference value and depth camera；
Step 2: can hold Kinect or Kinect is placed in mobile robot, be acquired to indoor scene, It includes a cromogram and depth map that wherein Kinect, which obtains each frame information, need SDK using Kinect by cromogram with Depth map alignment.
Step 3: tracking each frame image of input, this step contains the rejecting of dynamic point part and right The initial tracking of camera pose.
1, it extracts characteristic point: extracting the characteristic point in color image using ORB characteristic point algorithm.
2, dynamic point detection and rejecting:
A) image segmentation and identification: color image is split using the SegNet network of re -training, is obtained each The region of the label of pixel, obtained people and other animals belongs to dynamic area, in characteristic point belong to dynamic point, to these Dynamic point carries out rejecting operation.
B) track: characteristic point and previous frame or a upper reference frame after present frame is rejected do Feature Points Matching, will An initial estimation pose of present frame is estimated using rate pattern or reference frame model
C) multi-view geometry method judges dynamic point: Fig. 2 is that multi-view geometry method judges dynamic point schematic illustration；Using upper The initial estimation pose that one step acquires carries out characteristic point to judge whether to belong to dynamic point, and the principle specifically judged is as follows: will be empty Between point X coordinate projection to present frame on can obtain picture point x ', while can estimate its depth information zprojIf one A dynamic object blocks X, then the depth information z ' that actual measurement obtains can be less than zproj, as shown in Figure 2.Therefore work as Some point ' Δ z=zprojWhen-z ' is greater than some threshold tau, it is classified as dynamic point.By many experiments, τ is arranged for we For 0.5m.
D) dynamic point is obtained to multi-view geometry method, region growing operation is carried out on depth map to it, obtains dynamic area Domain.
E) two dynamic areas for obtaining convolutional neural networks and multi-view geometry method are merged: the specific behaviour of fusion It is that union is taken to two regions as method.
F) in order to improve detection speed, every 5 frame carries out a dynamic area detection, for not carrying out dynamic area detection Frame needs to carry out Feature Points Matching with the frame for having carried out dynamic area detection.Utilize the spy fallen in outside dynamic area after matching Sign point carry out camera pose estimation.
3, it tracks again: after further rejecting dynamic point using multi-view geometry method, re-using rate pattern and reference Frame model estimate present frame pose, is more accurately tracked in local map after obtaining initial pose.
4, be inserted into new key frame: the foundation that new key frame is inserted into judgement is: the last insertion key frame of distance has been more than 20 It is more than frame, present frame tracked to point map less than 50, the coincidence ratio of present frame and reference frame is less than 90%.
Step 4: local map optimization: to the key frame being newly added, new all frames and all total of regarding altogether that key frame is added BA optimization is carried out depending on the point map in frame.
If setting observational equation as z=h (ξ, p), ξ is the representation of Lie algebra of camera pose here, and p indicates the generation of road sign point Boundary's coordinate, and observing data z indicates pixel coordinate z=[us, vs]T, therefore projection error is expressed as e=z-h (ξ, p), for BA Optimization is to take into account total depending on frame and altogether depending on the point map in frame, therefore total error term is written as
Namely allow above-mentioned error term minimum, wherein eijIt indicates to observe the error of j-th of characteristic point in i-th of pose.
Step 5: winding detection judges whether that previous frame constitutes closed loop therewith using the bag of words of current key frame, if It is examined by closed loop consistency, then calculates the similarity transformation between closed loop frame, consecutive frame carries out closed loop correction.
The final reconstructed results of the present invention are: can completely restore indoor static scene, and export indoor scene Colour point clouds map is put and effectively eliminates the ghost of the dynamic objects such as people generation in cloud map.
The above is not limitation of the present invention, it is noted that those skilled in the art are come It says, under the premise of not departing from essential scope of the present invention, several variations, modifications, additions or substitutions can also be made, these improvement It also should be regarded as protection scope of the present invention with retouching.

Claims (6)

1. a kind of three-dimensional rebuilding method based on RGBD camera and convolutional neural networks, which is characterized in that three-dimensional rebuilding method institute Device mainly includes RGBD camera, the PC machine with GPU, the specific steps are as follows:
1) it, demarcates RGBD camera: obtaining the color camera internal reference value of RGBD camera and the internal reference value of depth camera and depth phase The transfer matrix of machine and color camera；
2), acquire scene image: each frame include a cromogram and depth map, using RGBD camera SDK by cromogram with Depth map alignment；
3) it, extracts characteristic point: extracting the characteristic point in color image using ORB characteristic point algorithm；
4), dynamic point detection is with rejecting: dynamic is carried out in the way of combining by convolutional neural networks and multi-view geometry method The rejecting of point；
5) it, tracks again: after further rejecting dynamic point using multi-view geometry method, re-using rate pattern and reference frame mould Type estimate present frame pose, is more accurately tracked in local map after obtaining initial pose；
6), be inserted into new key frame: the foundation that new key frame is inserted into judgement is, the last insertion key frame of distance be more than 20 frames with On, present frame tracked to point map less than 50, the coincidence ratio of present frame and reference frame is less than 90%；
7), local map optimizes: in the key frame being newly added, new all view frames altogether that key frame is added and all frames of view altogether Point map carries out BA optimization；
8), winding detects: judging whether that previous frame constitutes closed loop therewith using the bag of words of current key frame, if it is consistent to cross closed loop Property inspection, then calculate the similarity transformation between closed loop frame, consecutive frame carries out closed loop correction.
2. the three-dimensional rebuilding method according to claim 1 based on RGBD camera and convolutional neural networks, which is characterized in that In the step 4), dynamic point detection includes the following steps: with rejecting
A), image segmentation and identification: a kind of SegNet network (semantic segmentation based on convolutional neural networks of re -training is utilized Network)；
Color image is split, the label of each pixel is obtained, the region of obtained people and other animals belongs to dynamic area Domain, in characteristic point belong to dynamic point, rejecting operation is carried out to these dynamic points；
B), track: characteristic point and previous frame or a upper reference frame after present frame is rejected do Feature Points Matching, will utilize Rate pattern or reference frame model estimate an initial pose of present frame；
C), multi-view geometry method judges dynamic point: projecting to estimation of Depth value on present frame by comparing a certain space characteristics point Judged with the depth measurement of the point, is to think that the point is dynamic point when its difference is greater than certain threshold value；
D), dynamic point is obtained to multi-view geometry method, region growing operation is carried out on depth map to it, obtains dynamic area；
E), two dynamic areas for obtaining convolutional neural networks and multi-view geometry method are merged, the concrete operations of fusion Method is to take union to two regions.
F), as a further improvement, in order to improve detection speed, every 5 frame carries out a dynamic area detection, i.e., every 5 frame weight Multiple above a)-e) step.
3. the three-dimensional rebuilding method according to claim 2 based on RGBD camera and convolutional neural networks, which is characterized in that The principle specifically judged in the step c) is as follows: image will can be obtained on the coordinate projection to present frame of spatial point X Point x ', while can estimate its depth information zprojIf a dynamic object blocks X, actual measurement is obtained The depth information z ' arrived can be less than zproj, therefore when some point ' Δ z=zprojWhen-z ' is greater than some threshold tau, it is classified as Dynamic point.
4. the three-dimensional rebuilding method according to claim 3 based on RGBD camera and convolutional neural networks, which is characterized in that The τ τ value is 0.5m.
5. the three-dimensional rebuilding method according to claim 1 based on RGBD camera and convolutional neural networks, which is characterized in that Further include following steps before step 7) after the step 6): feeding builds figure line journey, builds figure line journey and is responsible for camera pose, color Chromatic graph, depth map generate point cloud chart.
6. the three-dimensional reconstruction side based on RGBD camera and convolutional neural networks described according to claim 1 or 2 or 3 or 4 or 5 Method, which is characterized in that the RGBD camera is KinectV2 (Microsoft's second generation 3D body-sensing video camera).
CN201910572096.1A 2019-06-28 2019-06-28 RGBD camera-based three-dimensional reconstruction method for indoor dynamic scene Active CN110349250B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910572096.1A CN110349250B (en) 2019-06-28 2019-06-28 RGBD camera-based three-dimensional reconstruction method for indoor dynamic scene

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910572096.1A CN110349250B (en) 2019-06-28 2019-06-28 RGBD camera-based three-dimensional reconstruction method for indoor dynamic scene

Publications (2)

Publication Number Publication Date
CN110349250A true CN110349250A (en) 2019-10-18
CN110349250B CN110349250B (en) 2020-12-22

Family

ID=68177197

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910572096.1A Active CN110349250B (en) 2019-06-28 2019-06-28 RGBD camera-based three-dimensional reconstruction method for indoor dynamic scene

Country Status (1)

Country Link
CN (1) CN110349250B (en)

Cited By (30)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111160107A (en) * 2019-12-05 2020-05-15 东南大学 Dynamic region detection method based on feature matching
CN111161318A (en) * 2019-12-30 2020-05-15 广东工业大学 Dynamic scene SLAM method based on YOLO algorithm and GMS feature matching
CN111390975A (en) * 2020-04-27 2020-07-10 浙江库科自动化科技有限公司 Inspection intelligent robot with air pipe removing function and inspection method thereof
CN111402336A (en) * 2020-03-23 2020-07-10 中国科学院自动化研究所 Semantic S L AM-based dynamic environment camera pose estimation and semantic map construction method
CN111462179A (en) * 2020-03-26 2020-07-28 北京百度网讯科技有限公司 Three-dimensional object tracking method and device and electronic equipment
CN111709982A (en) * 2020-05-22 2020-09-25 浙江四点灵机器人股份有限公司 Three-dimensional reconstruction method for dynamic environment
CN111724439A (en) * 2019-11-29 2020-09-29 中国科学院上海微系统与信息技术研究所 Visual positioning method and device in dynamic scene
CN111739037A (en) * 2020-07-31 2020-10-02 之江实验室 Semantic segmentation method for indoor scene RGB-D image
CN111914832A (en) * 2020-06-03 2020-11-10 华南理工大学 SLAM method of RGB-D camera in dynamic scene
CN112037261A (en) * 2020-09-03 2020-12-04 北京华捷艾米科技有限公司 Method and device for removing dynamic features of image
CN112037268A (en) * 2020-09-02 2020-12-04 中国科学技术大学 Environment sensing method based on probability transfer model in dynamic scene
CN112101160A (en) * 2020-09-04 2020-12-18 浙江大学 Binocular semantic SLAM method oriented to automatic driving scene
CN112220444A (en) * 2019-11-20 2021-01-15 北京健康有益科技有限公司 Pupil distance measuring method and device based on depth camera
CN112258618A (en) * 2020-11-04 2021-01-22 中国科学院空天信息创新研究院 Semantic mapping and positioning method based on fusion of prior laser point cloud and depth map
CN112435262A (en) * 2020-11-27 2021-03-02 广东电网有限责任公司肇庆供电局 Dynamic environment information detection method based on semantic segmentation network and multi-view geometry
CN112530014A (en) * 2020-12-18 2021-03-19 北京理工大学重庆创新中心 Multi-unmanned aerial vehicle indoor scene three-dimensional reconstruction method and device
CN112651357A (en) * 2020-12-30 2021-04-13 浙江商汤科技开发有限公司 Segmentation method of target object in image, three-dimensional reconstruction method and related device
CN112767409A (en) * 2019-11-05 2021-05-07 珠海格力电器股份有限公司 Image processing method and device before positioning, storage medium and computer equipment
CN112802186A (en) * 2021-01-27 2021-05-14 清华大学 Dynamic scene real-time three-dimensional reconstruction method based on binarization characteristic coding matching
CN112802053A (en) * 2021-01-27 2021-05-14 广东工业大学 Dynamic object detection method for dense mapping in dynamic environment
CN112907677A (en) * 2019-12-04 2021-06-04 杭州海康威视数字技术股份有限公司 Camera calibration method and device for single-frame image and storage medium
CN113447014A (en) * 2021-08-30 2021-09-28 深圳市大道智创科技有限公司 Indoor mobile robot, mapping method, positioning method, and mapping positioning device
CN113673524A (en) * 2021-07-05 2021-11-19 北京物资学院 Method and device for removing dynamic characteristic points of warehouse semi-structured environment
CN114565656A (en) * 2022-02-10 2022-05-31 北京箩筐时空数据技术有限公司 Camera pose prediction method and device, storage medium and computer equipment
WO2022110514A1 (en) * 2020-11-27 2022-06-02 叠境数字科技（上海）有限公司 Image interpolation method and apparatus employing rgb-d image and multi-camera system
CN114723672A (en) * 2022-03-09 2022-07-08 杭州易现先进科技有限公司 Method, system, device and medium for three-dimensional reconstruction data acquisition and verification
CN114972656A (en) * 2022-06-23 2022-08-30 安徽工业大学 Dynamic scene vision SLAM optimization method based on semantic segmentation network
CN116206068A (en) * 2023-04-28 2023-06-02 北京科技大学 Three-dimensional driving scene generation and construction method and device based on real data set
CN116258817A (en) * 2023-02-16 2023-06-13 浙江大学 Automatic driving digital twin scene construction method and system based on multi-view three-dimensional reconstruction
CN116452776A (en) * 2023-06-19 2023-07-18 国网浙江省电力有限公司湖州供电公司 Low-carbon substation scene reconstruction method based on vision synchronous positioning and mapping system

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103049929A (en) * 2012-11-20 2013-04-17 浙江大学 Multi-camera dynamic scene 3D (three-dimensional) rebuilding method based on joint optimization
US20130182894A1 (en) * 2012-01-18 2013-07-18 Samsung Electronics Co., Ltd. Method and apparatus for camera tracking
CN107193279A (en) * 2017-05-09 2017-09-22 复旦大学 Robot localization and map structuring system based on monocular vision and IMU information
CN107833236A (en) * 2017-10-31 2018-03-23 中国科学院电子学研究所 Semantic vision positioning system and method are combined under a kind of dynamic environment
CN108596974A (en) * 2018-04-04 2018-09-28 清华大学 Dynamic scene robot localization builds drawing system and method
KR20180112622A (en) * 2017-04-04 2018-10-12 엘지전자 주식회사 Method of configuring position based on identification of fixed object and moving object and robot implementing thereof
CN109387204A (en) * 2018-09-26 2019-02-26 东北大学 The synchronous positioning of the mobile robot of dynamic environment and patterning process in faced chamber

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130182894A1 (en) * 2012-01-18 2013-07-18 Samsung Electronics Co., Ltd. Method and apparatus for camera tracking
CN103049929A (en) * 2012-11-20 2013-04-17 浙江大学 Multi-camera dynamic scene 3D (three-dimensional) rebuilding method based on joint optimization
KR20180112622A (en) * 2017-04-04 2018-10-12 엘지전자 주식회사 Method of configuring position based on identification of fixed object and moving object and robot implementing thereof
CN107193279A (en) * 2017-05-09 2017-09-22 复旦大学 Robot localization and map structuring system based on monocular vision and IMU information
CN107833236A (en) * 2017-10-31 2018-03-23 中国科学院电子学研究所 Semantic vision positioning system and method are combined under a kind of dynamic environment
CN108596974A (en) * 2018-04-04 2018-09-28 清华大学 Dynamic scene robot localization builds drawing system and method
CN109387204A (en) * 2018-09-26 2019-02-26 东北大学 The synchronous positioning of the mobile robot of dynamic environment and patterning process in faced chamber

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
JIANBO ZHANG,ET AL: "《Semantic Segmentation based Dense RGB-D SLAM in Dynamic Environments》", 《2019 3RD INTERNATIONAL CONFERENCE ON ARTIFICIAL INTELLIGENCE, AUTOMATION AND CONTROL TECHNOLOGIES (AIACT 2019) 》 *
代具亭: "《基于RGB-D视频序列的大尺度场景三维语义表面重建技术研究》", 《中国博士学位论文全文数据库 信息科技辑》 *

Cited By (47)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112767409A (en) * 2019-11-05 2021-05-07 珠海格力电器股份有限公司 Image processing method and device before positioning, storage medium and computer equipment
CN112220444A (en) * 2019-11-20 2021-01-15 北京健康有益科技有限公司 Pupil distance measuring method and device based on depth camera
CN112220444B (en) * 2019-11-20 2021-06-29 北京健康有益科技有限公司 Pupil distance measuring method and device based on depth camera
CN111724439A (en) * 2019-11-29 2020-09-29 中国科学院上海微系统与信息技术研究所 Visual positioning method and device in dynamic scene
CN111724439B (en) * 2019-11-29 2024-05-17 中国科学院上海微系统与信息技术研究所 Visual positioning method and device under dynamic scene
CN112907677B (en) * 2019-12-04 2023-07-25 杭州海康威视数字技术股份有限公司 Camera calibration method and device for single-frame image and storage medium
CN112907677A (en) * 2019-12-04 2021-06-04 杭州海康威视数字技术股份有限公司 Camera calibration method and device for single-frame image and storage medium
CN111160107A (en) * 2019-12-05 2020-05-15 东南大学 Dynamic region detection method based on feature matching
CN111161318A (en) * 2019-12-30 2020-05-15 广东工业大学 Dynamic scene SLAM method based on YOLO algorithm and GMS feature matching
CN111402336B (en) * 2020-03-23 2024-03-12 中国科学院自动化研究所 Semantic SLAM-based dynamic environment camera pose estimation and semantic map construction method
CN111402336A (en) * 2020-03-23 2020-07-10 中国科学院自动化研究所 Semantic S L AM-based dynamic environment camera pose estimation and semantic map construction method
CN111462179A (en) * 2020-03-26 2020-07-28 北京百度网讯科技有限公司 Three-dimensional object tracking method and device and electronic equipment
CN111462179B (en) * 2020-03-26 2023-06-27 北京百度网讯科技有限公司 Three-dimensional object tracking method and device and electronic equipment
CN111390975A (en) * 2020-04-27 2020-07-10 浙江库科自动化科技有限公司 Inspection intelligent robot with air pipe removing function and inspection method thereof
CN111709982A (en) * 2020-05-22 2020-09-25 浙江四点灵机器人股份有限公司 Three-dimensional reconstruction method for dynamic environment
CN111914832B (en) * 2020-06-03 2023-06-13 华南理工大学 SLAM method of RGB-D camera under dynamic scene
CN111914832A (en) * 2020-06-03 2020-11-10 华南理工大学 SLAM method of RGB-D camera in dynamic scene
CN111739037A (en) * 2020-07-31 2020-10-02 之江实验室 Semantic segmentation method for indoor scene RGB-D image
CN111739037B (en) * 2020-07-31 2020-12-01 之江实验室 Semantic segmentation method for indoor scene RGB-D image
CN112037268B (en) * 2020-09-02 2022-09-02 中国科学技术大学 Environment sensing method based on probability transfer model in dynamic scene
CN112037268A (en) * 2020-09-02 2020-12-04 中国科学技术大学 Environment sensing method based on probability transfer model in dynamic scene
CN112037261A (en) * 2020-09-03 2020-12-04 北京华捷艾米科技有限公司 Method and device for removing dynamic features of image
CN112101160A (en) * 2020-09-04 2020-12-18 浙江大学 Binocular semantic SLAM method oriented to automatic driving scene
CN112101160B (en) * 2020-09-04 2024-01-05 浙江大学 Binocular semantic SLAM method for automatic driving scene
CN112258618A (en) * 2020-11-04 2021-01-22 中国科学院空天信息创新研究院 Semantic mapping and positioning method based on fusion of prior laser point cloud and depth map
CN112258618B (en) * 2020-11-04 2021-05-14 中国科学院空天信息创新研究院 Semantic mapping and positioning method based on fusion of prior laser point cloud and depth map
CN112435262A (en) * 2020-11-27 2021-03-02 广东电网有限责任公司肇庆供电局 Dynamic environment information detection method based on semantic segmentation network and multi-view geometry
WO2022110514A1 (en) * 2020-11-27 2022-06-02 叠境数字科技（上海）有限公司 Image interpolation method and apparatus employing rgb-d image and multi-camera system
CN112530014B (en) * 2020-12-18 2023-07-25 北京理工大学重庆创新中心 Three-dimensional reconstruction method and device for indoor scene of multiple unmanned aerial vehicles
CN112530014A (en) * 2020-12-18 2021-03-19 北京理工大学重庆创新中心 Multi-unmanned aerial vehicle indoor scene three-dimensional reconstruction method and device
CN112651357A (en) * 2020-12-30 2021-04-13 浙江商汤科技开发有限公司 Segmentation method of target object in image, three-dimensional reconstruction method and related device
CN112651357B (en) * 2020-12-30 2024-05-24 浙江商汤科技开发有限公司 Method for segmenting target object in image, three-dimensional reconstruction method and related device
CN112802186B (en) * 2021-01-27 2022-06-24 清华大学 Dynamic scene real-time three-dimensional reconstruction method based on binarization characteristic coding matching
CN112802053A (en) * 2021-01-27 2021-05-14 广东工业大学 Dynamic object detection method for dense mapping in dynamic environment
CN112802186A (en) * 2021-01-27 2021-05-14 清华大学 Dynamic scene real-time three-dimensional reconstruction method based on binarization characteristic coding matching
CN113673524A (en) * 2021-07-05 2021-11-19 北京物资学院 Method and device for removing dynamic characteristic points of warehouse semi-structured environment
CN113447014A (en) * 2021-08-30 2021-09-28 深圳市大道智创科技有限公司 Indoor mobile robot, mapping method, positioning method, and mapping positioning device
CN114565656A (en) * 2022-02-10 2022-05-31 北京箩筐时空数据技术有限公司 Camera pose prediction method and device, storage medium and computer equipment
CN114723672A (en) * 2022-03-09 2022-07-08 杭州易现先进科技有限公司 Method, system, device and medium for three-dimensional reconstruction data acquisition and verification
CN114723672B (en) * 2022-03-09 2024-08-20 杭州易现先进科技有限公司 Method, system, device and medium for three-dimensional reconstruction data acquisition and verification
CN114972656A (en) * 2022-06-23 2022-08-30 安徽工业大学 Dynamic scene vision SLAM optimization method based on semantic segmentation network
CN114972656B (en) * 2022-06-23 2024-08-16 安徽工业大学 Dynamic scene vision SLAM optimization method based on semantic segmentation network
CN116258817B (en) * 2023-02-16 2024-01-30 浙江大学 Automatic driving digital twin scene construction method and system based on multi-view three-dimensional reconstruction
CN116258817A (en) * 2023-02-16 2023-06-13 浙江大学 Automatic driving digital twin scene construction method and system based on multi-view three-dimensional reconstruction
CN116206068A (en) * 2023-04-28 2023-06-02 北京科技大学 Three-dimensional driving scene generation and construction method and device based on real data set
CN116452776B (en) * 2023-06-19 2023-10-20 国网浙江省电力有限公司湖州供电公司 Low-carbon substation scene reconstruction method based on vision synchronous positioning and mapping system
CN116452776A (en) * 2023-06-19 2023-07-18 国网浙江省电力有限公司湖州供电公司 Low-carbon substation scene reconstruction method based on vision synchronous positioning and mapping system

Also Published As

Publication number Publication date
CN110349250B (en) 2020-12-22

Similar Documents

Publication Publication Date Title
CN110349250A (en) A kind of three-dimensional rebuilding method of the indoor dynamic scene based on RGBD camera
Xiao et al. Dynamic-SLAM: Semantic monocular visual localization and mapping based on deep learning in dynamic environment
Yu et al. DS-SLAM: A semantic visual SLAM towards dynamic environments
Paz et al. Large-scale 6-DOF SLAM with stereo-in-hand
CN103735269B (en) A kind of height measurement method followed the tracks of based on video multi-target
CN110967009A (en) Navigation positioning and map construction method and device for transformer substation inspection robot
CN114782626A (en) Transformer substation scene mapping and positioning optimization method based on laser and vision fusion
CN109947093A (en) A kind of intelligent barrier avoiding algorithm based on binocular vision
CN105303518A (en) Region feature based video inter-frame splicing method
CN111998862A (en) Dense binocular SLAM method based on BNN
Li et al. Robust stereo visual slam for dynamic environments with moving object
Song et al. DGM-VINS: Visual–inertial SLAM for complex dynamic environments with joint geometry feature extraction and multiple object tracking
Lin et al. Contour-SLAM: A robust object-level SLAM based on contour alignment
Cui et al. A monocular ORB-SLAM in dynamic environments
Min et al. Coeb-slam: A robust vslam in dynamic environments combined object detection, epipolar geometry constraint, and blur filtering
Swadzba et al. Dynamic 3D scene analysis for acquiring articulated scene models
Wang et al. Real-time omnidirectional visual SLAM with semi-dense mapping
Zou et al. Learning motion field of LiDAR point cloud with convolutional networks
Cui et al. Direct-ORB-SLAM: Direct Monocular ORB-SLAM
Zhao et al. Object Detection-based Visual SLAM for Dynamic Scenes
Yao et al. Fast and robust visual odometry with a low-cost IMU in dynamic environments
Yang et al. A review of visual odometry in SLAM techniques
WO2022193193A1 (en) Data processing method and device
Shi et al. Dynamic Visual SLAM Based on Semantic Information and Multi-View Geometry
Yu et al. Visual SLAM algorithm based on ORB features and line features

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant