CN110349250A - A kind of three-dimensional rebuilding method of the indoor dynamic scene based on RGBD camera - Google Patents

Abstract

The invention discloses a three-dimensional reconstruction method of an indoor dynamic scene based on an RGBD camera, including the method of calibrating the RGBD camera, collecting scene images, extracting feature points, detecting and eliminating dynamic points, and combining convolutional neural network and multi-view geometric method To perform steps such as dynamic point elimination, re-tracking, insertion of new key frames, local map optimization, and loop detection. Using this method effectively solves the problem of inaccurate camera pose estimation and dynamic object ghosting in 3D reconstruction in dynamic scenes. , the present invention optimizes the time of the dynamic area detection step, and can realize the effect of real-time reconstruction. On the TUM dataset, we use the absolute error of the camera trajectory to measure, and its root mean square error has increased from the previous 0.752 to 0.025.

Description

A 3D reconstruction method of indoor dynamic scene based on RGBD camera

technical field

The invention belongs to the field of three-dimensional imaging, in particular, it is a three-dimensional reconstruction method of an indoor dynamic scene based on an RGBD camera.

Background technique

With the advancement of computer technology in recent years, AR and VR technology has gradually become one of the hot research areas. In the field of smart home, it has important applications in the field of virtual shopping. How to effectively reconstruct the surrounding scenes is the direction of research. one. With the advancement of artificial intelligence technology, in the field of automatic driving of cars, the field of autonomous flight of drones also needs to solve the construction of surrounding scenes and its own positioning.

The emergence of SLAM (simultaneous localization and map construction) technology has solved the above problems well. The SLAM system uses sensors mounted on objects (such as monocular and binocular cameras in visual SLAM) to obtain external information, uses this information to estimate its own position and attitude, and constructs maps as needed.

At the same time, in recent years, the performance of deep learning neural networks in the field of image recognition and segmentation has been amazing. Alex-Net, Google-Net, VGG, Res-Net and other networks have continuously refreshed the accuracy of target recognition datasets. Therefore, the semantic SLAM system combined with deep learning has a new development. The semantic SLAM system can recognize objects in the scene while reconstructing the surrounding scene. Its application in the field of robotics can improve the robot's understanding and cognition of the surrounding scene, and can provide the possibility for the robot to complete more complex tasks. Applying it in the field of automatic driving can realize active obstacle avoidance, danger warning and other functions, so semantic SLAM has broad application prospects.

Traditional SLAM methods solve the positioning and mapping problems in static scenes very well, but in scenes with dynamic objects, the positioning and mapping accuracy of these methods is very poor, because traditional SLAM methods are difficult to distinguish dynamic objects. objects and static objects, and they are treated the same. But in fact, the feature points extracted by the dynamic object are inconsistent with the feature points of the static background, which will seriously affect the positioning of the camera position, thereby affecting the result of the mapping. In practical applications, it is also necessary to eliminate dynamic objects. For example, in the path planning and navigation of sweeping robots, if dynamic objects such as people, dogs, and cats cannot be eliminated, the robot's navigation path will be deviated.

The invention solves the problems of inaccurate trajectory estimation and ghosting during map reconstruction in the SLAM system. The present invention adds a convolutional neural network to the SLAM system, uses the convolutional neural network to segment objects, and uses prior knowledge and multi-view geometry to judge dynamic points, thereby eliminating the impact of dynamic points on the scene.

Contents of the invention

The purpose of the present invention is to propose a three-dimensional reconstruction method based on RGBD cameras and convolutional neural networks for the problems existing in existing three-dimensional reconstruction methods in dynamic scenes. It uses depth cameras and color cameras to perform three-dimensional reconstruction of indoor scenes. The method of semantic segmentation eliminates the impact of dynamic objects on the final reconstruction effect.

The present invention is achieved through the following technical solutions:

The invention discloses a three-dimensional reconstruction method based on an RGBD camera and a convolutional neural network. The devices used in the three-dimensional reconstruction method mainly include an RGBD camera and a PC with a GPU. The specific steps are as follows:

1), Calibrate the RGBD camera: Obtain the internal parameter value of the color camera of the RGBD camera and the internal parameter value of the depth camera, as well as the transfer matrix of the depth camera and the color camera;

2) Collect scene images: each frame includes a color map and a depth map, and use the SDK of the RGBD camera to align the color map with the depth map;

3), extract feature points: use the ORB feature point algorithm to extract feature points in the color image;

4), dynamic point detection and elimination: use the combination of convolutional neural network and multi-view geometric method to eliminate dynamic points;

5) Re-tracking: After using the multi-view geometry method to further eliminate dynamic points, re-use the velocity model and the reference frame model to estimate the pose of the current frame, and perform more accurate tracking in the local map after obtaining the initial pose ;

6) Insert a new key frame: The basis for judging the insertion of a new key frame is that the distance from the last inserted key frame exceeds 20 frames, the number of map points tracked by the current frame is less than 50, and the overlap ratio between the current frame and the reference frame less than 90%;

7), local map optimization: perform BA optimization on newly added key frames, all common view frames of newly added key frames, and map points in all common view frames;

8) Loop closure detection: Use the bag-of-words model of the current key frame to determine whether it forms a closed loop with the previous frame. If it passes the closed-loop consistency check, calculate the similarity transformation between closed-loop frames, and perform closed-loop correction on adjacent frames.

As a further improvement, in step 4) of the present invention, dynamic point detection and elimination include the following steps:

a), image segmentation and recognition: use the retrained SegNet network (a semantic segmentation network based on convolutional neural network) to segment the color image, get the label of each pixel, and the obtained areas of people and other animals are dynamic The feature points in the area belong to dynamic points, and these dynamic points are eliminated;

b) Tracking: match the feature points of the current frame with the feature points of the previous frame or the previous reference frame, and use the velocity model or the reference frame model to estimate an initial pose of the current frame;

c) Judgment of dynamic points by multi-view geometry method: Judgment is made by comparing the estimated depth value of a certain spatial feature point projected onto the current frame with the measured depth value of the point. When the difference is greater than a certain threshold, the point is considered to be dynamic. point;

d) Obtain dynamic points from the multi-view geometry method, and perform region growing operations on the depth map to obtain dynamic regions;

e) Fusing the two dynamic regions obtained by the convolutional neural network and the multi-view geometry method, the specific operation method of the fusion is to take the union of the two regions.

f) As a further improvement, in order to increase the detection speed, the dynamic region detection is performed every 5 frames, that is, the above steps a)-e) are repeated every 5 frames.

As a further improvement, the principle of the specific judgment in step c) of the present invention is as follows: the coordinates of the spatial point X can be projected onto the current frame to obtain the image point x′, and its depth information z _proj can be estimated at the same time, if If a dynamic object blocks X, the actual measured depth information z′ will be smaller than z _proj , so when a point’Δz=z _proj -z′ is greater than a certain threshold τ, it is classified as dynamic point.

As a further improvement, the value of τ described in the present invention is 0.5m.

As a further improvement, after step 6) described in the present invention, before step 7) also includes the following steps: send into the mapping thread, and the mapping thread is responsible for generating the point cloud image with the camera pose, color map, and depth map.

As a further improvement, the RGBD camera described in the present invention is Kinect V2 (Microsoft's second generation 3D somatosensory camera).

The beneficial effects of the present invention are as follows:

The present invention proposes a 3D reconstruction method combined with semantic segmentation, which effectively solves the problems of inaccurate camera pose estimation and ghosting of dynamic objects in 3D reconstruction in dynamic scenes, and uses the SegNet network to segment images, Use prior knowledge to exclude points belonging to the dynamic area, and then use the multi-view geometry method to further judge other points belonging to the dynamic part of the image, and finally use all static points for camera pose estimation and 3D scene reconstruction. The invention optimizes the time of the dynamic area detection step, and can realize the effect of real-time reconstruction. On the TUM dataset, we use the absolute error of the camera trajectory to measure, and its root mean square error has increased from the previous 0.752 to 0.025.

Description of drawings

Fig. 1 is a schematic flow chart of the system of the present invention;

Fig. 2 is a schematic diagram of the principle of judging dynamic points by the multi-view geometry method.

Specific implementation method

The present invention discloses an indoor three-dimensional reconstruction method based on RGBD camera, which focuses on solving dynamic objects in indoor scenes. The device is mainly composed of RGBD camera and PC with GPU. The RGBD camera is Kinect. Figure 1 is the system of the present invention Schematic diagram of the process; the specific steps are as follows:

Calibrate the Kinect camera, get the Kinect color camera internal parameter value and depth camera internal parameter value and the transfer matrix of the depth camera and color camera.

Collect scene images: each frame includes a color map and a depth map, and use the Kinect SDK to align the color map with the depth map.

Extract feature points: use the ORB feature point algorithm to extract feature points in the color image.

Dynamic point elimination: The present invention utilizes a combination of convolutional neural network and multi-view geometry method to eliminate dynamic points. The specific method is as follows:

1. Use the retrained SegNet network to segment the color image to obtain the label of each pixel. The obtained areas of people and other animals belong to the dynamic area, and the feature points in it belong to the dynamic points, and these dynamic points are eliminated.

2. Tracking: Match the feature points of the current frame with the feature points of the previous frame or the previous reference frame, and use the velocity model or reference frame model to estimate an initial pose of the current frame

3. Judgment of dynamic points by multi-view geometry method: Judgment is made by comparing the estimated depth value of a certain spatial feature point projected on the current frame with the measured depth value of the point. When the difference is greater than a certain threshold, the point is considered to be a dynamic point .

4. The dynamic point is obtained by the multi-view geometry method, and the region growing operation is performed on it on the depth map to obtain the dynamic region.

5. Fuse the two dynamic regions obtained by the convolutional neural network and the multi-view geometry method: the specific operation method of the fusion is to take the union of the two regions.

6. In order to improve the detection speed, a dynamic area detection is performed every 5 frames.

Re-tracking: After removing dynamic points, re-use the velocity model and reference frame model to estimate the pose of the current frame, and perform more accurate tracking in the local map after obtaining the initial pose.

Insert a new key frame: The basis for judging the insertion of a new key frame is: the distance from the last inserted key frame exceeds 20 frames, the number of map points tracked by the current frame is less than 50, and the overlap ratio between the current frame and the reference frame is less than 90 %.

Send it to the mapping thread, which is responsible for generating a point cloud image from the camera pose, color map, and depth map.

Local map optimization: For newly added key frames, find out their common view frames and map points that can be observed by these common view frames, and perform BA optimization.

Loop detection: Use the bag-of-words model of the current key frame to judge whether it can form a closed loop with the previous frame. If it passes the closed-loop consistency check, calculate the similarity transformation between closed-loop frames, and perform closed-loop correction on adjacent frames.

The technical scheme of the present invention will be further described below by specific examples:

As a preferred RGBD camera, KinectV2 is adopted, and the specific steps are as follows:

Step 1: Use the checkerboard to calibrate KinectV2, and get Kinect's internal parameter values of the color camera and depth camera and the transfer matrix R, T of the depth camera and color camera through the calibration algorithm;

Step 2: You can hold the Kinect or put the Kinect on the mobile robot to collect the indoor scene. Kinect gets each frame information including a color map and a depth map. You need to use the Kinect SDK to align the color map with the depth map.

Step 3: Track each frame of the input image. This step includes the removal of dynamic points and the initial tracking of the camera pose.

1. Extract feature points: Use the ORB feature point algorithm to extract feature points in the color image.

2. Dynamic point detection and elimination:

a) Image segmentation and recognition: Use the retrained SegNet network to segment the color image to get the label of each pixel. The obtained areas of people and other animals belong to the dynamic area, and the feature points in it belong to the dynamic points. For these dynamic Click to delete.

b) Tracking: match the feature points of the current frame with the feature points of the previous frame or the previous reference frame, and use the velocity model or the reference frame model to estimate an initial estimated pose of the current frame

c) Multi-view geometry method to judge dynamic points: Figure 2 is a schematic diagram of the principle of multi-view geometry method to judge dynamic points; use the initial estimated pose obtained in the previous step to judge whether a feature point belongs to a dynamic point, and the specific judgment principle is as follows: The coordinates of the spatial point X are projected onto the current frame to obtain the image point x′, and its depth information z _proj can be estimated at the same time. If a dynamic object blocks X, the actual measured depth information z′ will be smaller than z _proj ,as shown in picture 2. Therefore, when 'Δz=z _proj -z' of a certain point is greater than a certain threshold τ, it is classified as a dynamic point. After many experiments we set τ as 0.5m.

d) The multi-view geometry method is used to obtain dynamic points, and the region growing operation is performed on the depth map to obtain dynamic regions.

e) Fuse the two dynamic regions obtained by the convolutional neural network and the multi-view geometry method: the specific operation method of fusion is to take the union of the two regions.

f) In order to improve the detection speed, a dynamic area detection is performed every 5 frames. For frames without dynamic area detection, it is necessary to perform feature point matching with frames that have undergone dynamic area detection. The camera pose estimation is performed using the matched feature points that fall outside the dynamic region.

3. Re-tracking: After using the multi-view geometry method to further eliminate dynamic points, re-use the velocity model and the reference frame model to estimate the pose of the current frame, and perform more accurate tracking in the local map after obtaining the initial pose.

4. Insert a new key frame: The basis for judging the insertion of a new key frame is: the distance from the last inserted key frame exceeds 20 frames, the number of map points tracked by the current frame is less than 50, and the overlap ratio between the current frame and the reference frame is small at 90%.

Step 4: Local map optimization: BA optimization is performed on the newly added key frame, all common view frames of the newly added key frame, and map points in all common view frames.

If the observation equation is set as z=h(ξ, p), where ξ is the Lie algebra representation of the camera pose, p represents the world coordinates of landmark points, and the observation data z represents the pixel coordinates z=[u _s , v _s ] ^T , so the projection error is expressed as e=zh(ξ, p), for BA optimization is to take the common-view frame and the map points in the common-view frame into account, so the total error term is written as

That is to minimize the above error term, where e _ij represents the error of observing the j-th feature point at the i-th pose.

Step 5: Loop closure detection. Use the bag-of-words model of the current key frame to determine whether it forms a closed loop with the previous frame. If it passes the closed-loop consistency check, calculate the similarity transformation between closed-loop frames, and perform closed-loop correction on adjacent frames.

The final reconstruction result of the present invention is: the indoor static scene can be fully restored, and the colored point cloud map of the indoor scene can be output, and the afterimages produced by dynamic objects such as people are effectively eliminated in the point cloud map.

The above is not a limitation of the present invention. It should be pointed out that for those of ordinary skill in the art, some changes, modifications, additions or substitutions can be made without departing from the essential scope of the present invention. These Improvements and retouches should also be considered within the protection scope of the present invention.

Claims (6)

1. a kind of three-dimensional rebuilding method based on RGBD camera and convolutional neural networks, which is characterized in that three-dimensional rebuilding method institute Device mainly includes RGBD camera, the PC machine with GPU, the specific steps are as follows:

1) it, demarcates RGBD camera: obtaining the color camera internal reference value of RGBD camera and the internal reference value of depth camera and depth phase The transfer matrix of machine and color camera；

2), acquire scene image: each frame include a cromogram and depth map, using RGBD camera SDK by cromogram with Depth map alignment；

3) it, extracts characteristic point: extracting the characteristic point in color image using ORB characteristic point algorithm；

4), dynamic point detection is with rejecting: dynamic is carried out in the way of combining by convolutional neural networks and multi-view geometry method The rejecting of point；

5) it, tracks again: after further rejecting dynamic point using multi-view geometry method, re-using rate pattern and reference frame mould Type estimate present frame pose, is more accurately tracked in local map after obtaining initial pose；

6), be inserted into new key frame: the foundation that new key frame is inserted into judgement is, the last insertion key frame of distance be more than 20 frames with On, present frame tracked to point map less than 50, the coincidence ratio of present frame and reference frame is less than 90%；

7), local map optimizes: in the key frame being newly added, new all view frames altogether that key frame is added and all frames of view altogether Point map carries out BA optimization；

8), winding detects: judging whether that previous frame constitutes closed loop therewith using the bag of words of current key frame, if it is consistent to cross closed loop Property inspection, then calculate the similarity transformation between closed loop frame, consecutive frame carries out closed loop correction.

2. the three-dimensional rebuilding method according to claim 1 based on RGBD camera and convolutional neural networks, which is characterized in that In the step 4), dynamic point detection includes the following steps: with rejecting

A), image segmentation and identification: a kind of SegNet network (semantic segmentation based on convolutional neural networks of re -training is utilized Network)；

Color image is split, the label of each pixel is obtained, the region of obtained people and other animals belongs to dynamic area Domain, in characteristic point belong to dynamic point, rejecting operation is carried out to these dynamic points；

B), track: characteristic point and previous frame or a upper reference frame after present frame is rejected do Feature Points Matching, will utilize Rate pattern or reference frame model estimate an initial pose of present frame；

C), multi-view geometry method judges dynamic point: projecting to estimation of Depth value on present frame by comparing a certain space characteristics point Judged with the depth measurement of the point, is to think that the point is dynamic point when its difference is greater than certain threshold value；

D), dynamic point is obtained to multi-view geometry method, region growing operation is carried out on depth map to it, obtains dynamic area；

E), two dynamic areas for obtaining convolutional neural networks and multi-view geometry method are merged, the concrete operations of fusion Method is to take union to two regions.

F), as a further improvement, in order to improve detection speed, every 5 frame carries out a dynamic area detection, i.e., every 5 frame weight Multiple above a)-e) step.

3. the three-dimensional rebuilding method according to claim 2 based on RGBD camera and convolutional neural networks, which is characterized in that The principle specifically judged in the step c) is as follows: image will can be obtained on the coordinate projection to present frame of spatial point X Point x ', while can estimate its depth information z_projIf a dynamic object blocks X, actual measurement is obtained The depth information z ' arrived can be less than z_proj, therefore when some point ' Δ z=z_projWhen-z ' is greater than some threshold tau, it is classified as Dynamic point.

4. the three-dimensional rebuilding method according to claim 3 based on RGBD camera and convolutional neural networks, which is characterized in that The τ τ value is 0.5m.

5. the three-dimensional rebuilding method according to claim 1 based on RGBD camera and convolutional neural networks, which is characterized in that Further include following steps before step 7) after the step 6): feeding builds figure line journey, builds figure line journey and is responsible for camera pose, color Chromatic graph, depth map generate point cloud chart.

6. the three-dimensional reconstruction side based on RGBD camera and convolutional neural networks described according to claim 1 or 2 or 3 or 4 or 5 Method, which is characterized in that the RGBD camera is KinectV2 (Microsoft's second generation 3D body-sensing video camera).