CN110243370A - A 3D Semantic Map Construction Method for Indoor Environment Based on Deep Learning - Google Patents

Abstract

A method for building a three-dimensional semantic map of an indoor environment based on deep learning, the steps are: 1) using a Kinect depth camera carried by a mobile robot to obtain an RGB-D image sequence of an indoor environment target scene; 2) using a trained RGB-D image sequence The semantic segmentation network of the image performs feature extraction and processing on the currently acquired RGB-D image; 3) Estimates the corresponding robot pose information P _t according to each frame of the input image; 4) Real-time relocation and closed-loop detection algorithm according to Randomized ferns Optimize the robot pose; 5) Construct a point cloud map with key frames, and fuse the point cloud corresponding to the newly acquired image frame with the constructed point cloud map; 6) Map the pixel-level semantic annotation results of the key frames to the corresponding points On the cloud map; 7) Optimize the semantic labeling information of the constructed 3D point cloud map by using the semantic tags of the key frames to obtain the 3D semantic map of the indoor environment; complete the real-time construction task of the indoor environment semantic map and improve the intelligence of the mobile robot's environmental perception Level.

Description

A 3D Semantic Map Construction Method for Indoor Environment Based on Deep Learning

technical field

The invention belongs to the technical field of indoor navigation of mobile robots, and in particular relates to a method for constructing a three-dimensional semantic map of an indoor environment based on deep learning.

Background technique

The construction of the target scene map is an important research content of the autonomous navigation of the mobile robot. Semantic labeling of the point cloud of the constructed map to generate a high-precision point cloud map with semantic information has important application value for the intelligent navigation of the mobile robot in the unknown environment. Mobile robots can communicate with users naturally through semantic maps, so as to complete human-computer interaction tasks such as autonomous driving and home services.

When the environment information of the mobile robot is completely unknown, there is no prior information about the environment and its own position, which requires the mobile robot to obtain relevant information about the environment through its own sensors during the movement process. In order to complete the construction of the environmental map and locate its own position in the map, this is the simultaneous localization and map construction technology (Simultaneous Localization and Mapping, SLAM). The existing scene map construction method is to extract and match the feature points of the target scene image sequence, and then obtain the sparse point cloud or road sign map of the target scene. Interactive tasks.

Scene maps with high-level semantic information enable robots to identify and model objects in space, and understand unknown environmental scene information more fully, thereby laying the foundation for more advanced human-computer interaction and more complex tasks. The traditional point cloud map labeling task relies on environmental geometric information or user-guided labels. Point cloud labeling is not accurate enough and needs to be done offline. With the rapid development of deep learning technology in the field of image perception, especially the convolutional neural network (Convolutional Neural Network, CNN) in image classification, a large number of scholars began to apply deep learning to image semantic segmentation, and then provide accurate pixel-level semantic annotation for point cloud maps, and realize the task of real-time construction of high-precision point cloud maps for indoor environments. Therefore, the research on indoor scene semantic map construction technology based on deep learning has important theoretical significance and broad application prospects.

Contents of the invention

In order to overcome the above-mentioned prior art problems, the object of the present invention is to provide a method for building a three-dimensional semantic map of an indoor environment based on deep learning; the present invention applies deep learning to the three-dimensional semantic map construction algorithm for an indoor environment, and can analyze three-dimensional point clouds Perform real-time and accurate pixel-level semantic annotation to build a 3D semantic map of the indoor environment in real time. It has the characteristics of implementation, accuracy and no need to go offline.

In order to achieve the above object, the technical scheme adopted in the present invention is:

A method for building a semantic map of an indoor environment based on deep learning, comprising the following specific steps:

Step 1, using the Kinect depth camera carried by the mobile robot itself to obtain the RGB-D image sequence of the indoor environment target scene;

Step 2, using a semantic segmentation network based on RGB-D images to perform feature extraction and processing for each frame of RGB-D images acquired;

Step 3. Estimate the corresponding robot pose information P _t according to each input frame image;

Step 4, optimize the pose of the robot according to the Randomized ferns real-time relocation and closed-loop detection algorithm;

Step 5, use the key frame to construct the point cloud map, and fuse the point cloud corresponding to the newly acquired image frame with the constructed point cloud map;

Step 6, map the pixel-level semantic labeling results of the key frame to the corresponding point cloud map, and obtain the semantic label of the key frame;

Step 7, use the newly acquired key frames to optimize the semantic annotation information of the constructed 3D point cloud map.

Step 2 uses the semantic segmentation network based on RGB-D images to extract and process the features of each acquired RGB-D image. Four input channels for pixel-level semantic prediction for each frame of input image.

In step 3, the specific method of estimating the corresponding robot pose information P _t according to each input frame image is: comprehensively utilizing the geometric pose estimation of the depth image and the photometric pose estimation of the RGB image, by minimizing the point plane error and photometric error to obtain the robot pose P _t ,

Point error:

in, is the kth vertex of the current frame depth image; v ^k and n ^k are the corresponding vertex and normal of the previous frame image respectively; T is the current pose transformation matrix;

Photometric error:

in, is the gray value of the current frame RGB image at point u; Indicates the gray value of a point u in the current frame RGB image projected on the previous frame RGB image;

Joint loss function: E _track = E _icp +0.1E _rgb

The updated robot pose transformation matrix is obtained by using Gauss-Newton nonlinear least squares: That is, after the update, the corresponding pose of the current frame is P _t = T′P _t-1 , and then according to the robot pose relationship between adjacent image frames, the key frame sequence for point cloud map construction is determined.

In step 4, the specific method of optimizing the pose of the robot according to the Randomized ferns real-time relocation and closed-loop detection algorithm is: encode each frame of the input image, calculate the similarity between image frames according to the encoded value, and then judge whether it is based on the similarity. Add a new key frame, and solve the similar transformation matrix for the new key frame to perform loop closure detection.

In step 5, the specific method of using the key frame to construct the point cloud map, and fusing the point cloud corresponding to the newly acquired image frame with the constructed point cloud map is: perform coordinate transformation on the point cloud corresponding to all depth images, So that the subsequent point cloud is in the same coordinate system as the first frame point cloud; find the best transformation relationship between each coherent overlapping point cloud, and accumulate these transformation relations to all point clouds, you can gradually The current point cloud is merged into the constructed point cloud map.

In step 6, the specific method of mapping the pixel-level semantic annotation results of the key frame to the corresponding point cloud map is as follows: according to the robot pose transformation matrix T _WC , the camera coordinates of each pixel point are converted into world coordinates, and finally According to the three-dimensional space coordinates corresponding to each pixel point, the two-dimensional semantic segmentation results of the key frame image are mapped to the corresponding three-dimensional point cloud map, and the semantic labeling task of the three-dimensional point cloud map is completed.

The closed-loop detection includes global closed-loop detection and local closed-loop detection, wherein, the node parameter optimization equation under the global closed-loop constraint is:

Among them, H represents the similarity transformation matrix, P _t represents the robot pose corresponding to the current frame image, Indicates the projection of point u in the depth map of the current frame, represents the initial pose of the robot, represents the initial moment;

The node parameter optimization equation under local closed-loop constraints is:

in, Represents the constraint on the deformation between the map built in the most recent time period and the map model built in the previous time period.

The specific method of step 7 is: initialize the point cloud label probability distribution with the semantic segmentation result of the key frame, and then use recursive Bayesian to update the point cloud label distribution probability:

Among them, c _t represents the category probability distribution of the point cloud at time t, Represents a set of key frames {K ₀ , K ₁ ,...,K _t }, Z represents a normalization constant, and K _t represents the key frame at time t;

The final semantic label for each point cloud can be obtained by maximizing this probability distribution function:

L(P)=argmaxP(c|K).

The beneficial effects of the present invention are:

1) The present invention comprehensively utilizes the color features of the RGB image and the geometric features of the depth image to improve the performance of the image semantic segmentation network, and reasonably deletes the network parameters through model compression. Accurate semantic segmentation results can be obtained quickly.

2) The present invention directly uses the Kinect depth camera carried by the robot itself to construct a map of the indoor environment, which can carry out real-time semantic annotation on the point cloud, and incrementally construct the semantic map of the indoor environment space, so that the mobile robot can map the indoor global semantic map. Intelligent navigation lays the foundation for the completion of human-computer interaction tasks such as automatic driving and home services.

Description of drawings

Fig. 1 is the general block diagram of the system of the present invention.

Fig. 2 is a block diagram of the ICNet semantic segmentation network structure based on RGB-D images of the present invention.

Figure 3 is a schematic diagram of semantic segmentation network model compression.

Figure 4 is a schematic diagram of the robot pose solution process.

Fig. 5 is a block diagram of the real-time relocation and closed-loop detection algorithm of Randomized ferns.

Figure 6(a) is a schematic diagram of robot pose optimization in the presence of a global closed-loop scenario.

Figure 6(b) is a schematic diagram of robot pose optimization without the idea of a global closed-loop situation.

Fig. 7 is a flow chart of point cloud fusion.

Fig. 8 is a flowchart of the present invention.

Detailed ways

The embodiments of the present invention will be further described below in conjunction with the accompanying drawings and specific implementation details:

Referring to Figures 1 and 8, a method for building a semantic map of an indoor environment based on deep learning includes the following steps:

Step 1, using the Kinect depth camera carried by the mobile robot itself to collect the RGB-D image sequence of the indoor environment target scene;

Step 2, use the RGB-D image-based semantic segmentation network to extract and process the features of each acquired RGB-D image; the overall block diagram of the RGB-D image-based semantic segmentation network is shown in Figure 2, and the design steps are as follows: :

1) In the Linux operating system, use the caffe deep learning framework to construct the ICNet semantic segmentation network based on RGB-D images, and stitch the RGB image and the depth image through the Concat layer as the four input channels of the semantic segmentation network;

2) Use the NYUD V2 indoor image standard dataset to train the image semantic segmentation network, divide the input RGB-D image pair into 1/4, 1/2 and full resolution images, and use them as the input of the three branch networks respectively;

3) After two feature map fusions, the convolution feature maps obtained from input images with three different resolutions are fused through the Eltwise layer;

4) The final fusion feature map undergoes multiple upsampling operations to restore the image resolution to the original input size, and finally obtain accurate semantic segmentation results;

The feature extraction and processing of depth images increases the number of parameters of the network model. In order to ensure the rapidity of the semantic segmentation network, model compression is required. In the process of network performance testing, the network parameters are reasonably deleted according to the L1 norm _of each convolution kernel, so as to achieve the purpose of quickly obtaining the semantic segmentation results of the input image. The specific process is shown in Figure 3.

Step 3, comprehensively utilize the geometric pose estimation of the depth image and the photometric pose estimation of the RGB image, and obtain the robot pose P _t by minimizing the point-plane error and the photometric error;

Among them, the rotation matrix R _t ∈ SO ₃ , the translation matrix t _t ∈ R ³ ,

Point error:

in, is the kth vertex of the current frame depth image; v ^k and n ^k are the corresponding vertex and normal of the previous frame image respectively; T is the current pose transformation matrix;

Photometric error:

in, is the gray value of the current frame RGB image at point u; Indicates the gray value of a point u in the current frame RGB image projected on the previous frame RGB image;

Joint loss function: E _track = E _icp +0.1E _rgb

The updated pose transformation matrix is obtained by the Gauss-Newton nonlinear least square method: That is, after the update, the current frame corresponds to the camera pose P _t = T′P _t-1 , the specific solution process is shown in Figure 4, and then according to the robot pose relationship between adjacent image frames, determine the key point cloud map construction frame sequence;

Step 4. Optimize the robot pose and point cloud map according to the randomized ferns real-time relocation and closed-loop detection algorithm. The overall process is shown in Figure 5. Randomized ferns encodes each frame of the input image and uses a special encoding storage method , to speed up the efficiency of image similarity comparison, the encoding method is as follows: Indicates that the encoding of each frame of image consists of m block encodings,

in, Indicates that each block code consists of n Ferns,

in, Indicates that each Fern determines the encoding of Ferns by comparing the pixel value at the pixel point x of the c channel with the threshold θ,

Calculate the block encoding for each newly acquired image, use the function Fers::generateFerns() to randomly initialize the position, channel and threshold θ of Ferns, and use the function Ferns::addFrame() to compare the new input image frame with the previous one according to the block encoding The similarity of the image frames is compared, and then the similarity is used to determine whether there are new key frames and closed loops;

If there is a global closed loop, as shown in Figure 6(a), use the tracking algorithm in step 1 to calculate the pose between the current frame and the i-th frame, and after obtaining the pose transformation matrix, uniformly sample the image and establish constraints, Optimize the node parameters, that is, the robot pose;

The node parameter optimization equation under global closed-loop constraints:

Among them, H represents the similarity transformation matrix, P _t represents the robot pose corresponding to the current frame image, Indicates the projection point of the current frame depth map in the camera coordinate system, represents the initial pose of the robot, represents the initial moment;

If there is no global closed loop, as shown in Figure 6(b), perform pose estimation on the local closed loop, and establish constraints to optimize node parameters;

Node parameter optimization equation under local closed-loop constraints:

in, Represents the constraint of deformation between the map constructed in the latest period and the map model constructed in the previous period;

Step 5, using OpenGL to fuse and update the point cloud, the specific process is shown in Figure 7. First, convert the 3D coordinates of each input vertex into 2D coordinates, calculate the color value of each vertex according to the lighting formula, and generate texture coordinates; then, combine the vertices processed in the first step with multiple vertices stored in the geometry shader The primitives are organized, clipped and rasterized; finally, the fragment shader is used to calculate the final color and depth value of the independent fragments generated after rasterization, and then spliced into a global point cloud map;

Step 6: While generating the global 3D point cloud map, perform semantic annotation, and map the pixel-level semantic annotation results of key frames to the corresponding point cloud map. According to the robot pose transformation matrix T _WC , each pixel can be The camera coordinates are converted into world coordinates, and finally, according to the 3D space coordinates corresponding to each pixel point, the 2D semantic segmentation results of the key frame image are mapped to the corresponding 3D point cloud map, and the semantic labeling task of the 3D point cloud map is completed ;

Step 7, since the newly acquired image frame will assign different labels to the point cloud, it is also necessary to optimize the semantic information of the constructed point cloud map according to the semantic label of the newly acquired key frame, and use the semantic segmentation result of the key frame to classify the point cloud The cloud label probability distribution is initialized, and the recursive Bayesian is used to update the point cloud label distribution probability:

Among them, c _t represents the category probability distribution of the point cloud at time t, Represents a set of key frames {K ₀ , K ₁ ,...,K _t }, Z represents a normalization constant, and K _t represents the key frame at time t; the final semantics of each point cloud can be obtained by maximizing the probability distribution function Label:

L(P)＝argmaxP(c|K)

where P(c|K) denotes the label probability distribution of the point cloud in the keyframe, and L(P) denotes the final semantic category of the point cloud.

Claims (10)

1. A method for building a three-dimensional semantic map of indoor environment based on deep learning, characterized in that, comprising the following steps: Step 1, using the Kinect depth camera carried by the mobile robot itself to obtain the RGB-D image sequence of the indoor environment target scene; Step 2, using a semantic segmentation network based on RGB-D images to perform feature extraction and processing for each frame of RGB-D images acquired; Step 3. Estimate the corresponding robot pose information P _t according to each input frame image; Step 4, optimize the pose of the robot according to the Randomized ferns real-time relocation and closed-loop detection algorithm; Step 5, use the key frame to construct the point cloud map, and fuse the point cloud corresponding to the newly acquired image frame with the constructed point cloud map; Step 6, map the pixel-level semantic labeling results of the key frame to the corresponding point cloud map, and obtain the semantic label of the key frame; Step 7: Use the semantic labels of the newly acquired key frames to optimize the semantic annotation information of the constructed 3D point cloud map. 2. a kind of indoor environment semantic map construction method based on deep learning according to claim 1, is characterized in that, in step 2, described adopting the semantic segmentation network based on RGB-D image to obtain each frame RGB-D Image feature extraction and processing, the specific method is: use the image cascade network ICNet, and use the depth information of the image as the fourth input channel of the network, and perform pixel-level semantic prediction for each frame of input image. 3. a kind of indoor environment semantic map construction method based on deep learning according to claim 1, is characterized in that, in step 3, the specific method of the corresponding robot pose information P _t estimated according to each frame image of input To: comprehensively utilize the geometric pose estimation of the depth image and the photometric pose estimation of the RGB image, obtain the robot pose P _t by minimizing the point-plane error and the photometric error, and then according to the robot pose relationship between adjacent image frames, Determine the sequence of keyframes for point cloud map construction. 4. a kind of indoor environment semantic map construction method based on deep learning according to claim 1, it is characterized in that, in step 4, the concrete method of described real-time relocation and closed-loop detection algorithm optimization robot pose according to Randomized ferns is: Encode each frame of input image, calculate the similarity between image frames according to the encoded value, and then judge whether to add a new key frame according to the similarity, and solve the similarity transformation matrix for the new key frame to perform closed-loop detection. 5. a kind of indoor environment semantic map construction method based on deep learning according to claim 1, it is characterized in that, in step 5, described utilize key frame to carry out point cloud map construction, and new acquisition image frame corresponding point cloud and The specific method of fusion of the constructed point cloud map is: coordinate transformation of the point cloud corresponding to all depth images, so that the subsequent point cloud is in the same coordinate system as the first frame point cloud; Find the best transformation relationship between the point clouds, and accumulate these transformation relations to all point clouds, you can gradually integrate the current point cloud into the reconstructed point cloud map. 6. A method for constructing an indoor environment semantic map based on deep learning according to claim 1, wherein in step 6, the specific method for mapping the pixel-level semantic labeling result of the key frame to the corresponding point cloud map is : According to the robot pose transformation matrix T _WC , the camera coordinates of each pixel are transformed into world coordinates, and finally, according to the 3D space coordinates corresponding to each pixel, the 2D semantic segmentation results of the key frame images are mapped to the corresponding On the 3D point cloud map of , complete the semantic annotation task of the 3D point cloud map. 7. a kind of indoor environment semantic map construction method based on deep learning according to claim 1, it is characterized in that, in step 7, described utilizes the semantic label optimization of newly acquired key frame to optimize the semantic annotation information that has built three-dimensional point cloud map The specific method is: initialize the point cloud label probability distribution with the semantic segmentation results of key frames, and then use recursive Bayesian to update the point cloud label distribution probability, and the final semantics of each point cloud can be obtained by maximizing the probability distribution function Label. 8. A deep learning-based indoor environment semantic map construction method according to claim 3, characterized in that the robot pose P _t is obtained by minimizing the point-plane error and photometric error described in step 3, and its specific method Yes: Point error: in, is the kth vertex of the current frame depth image; v ^k and n ^k are the corresponding vertex and normal of the previous frame image respectively; T is the current pose transformation matrix; Photometric error: in, is the gray value of the current frame RGB image at point u; Indicates the gray value of a point u in the current frame RGB image projected on the previous frame RGB image; Joint loss function: E _track = E _icp +0.1E _rgb The updated robot pose transformation matrix is obtained by using Gauss-Newton nonlinear least squares: That is, after the update, the corresponding pose of the current frame is P _t =T′P _t-1 . 9. A method for constructing an indoor environment semantic map based on deep learning according to claim 4, wherein said closed-loop detection includes global closed-loop detection and local closed-loop detection, wherein the node parameter optimization equation under global closed-loop constraints for: Among them, H represents the similarity transformation matrix, P _t represents the robot pose corresponding to the current frame image, Indicates the projection of point u in the depth map of the current frame, represents the initial pose of the robot, represents the initial moment; The node parameter optimization equation under local closed-loop constraints is: in, Represents the constraint on the deformation between the map built in the most recent time period and the map model built in the previous time period. 10. A kind of indoor environment semantic map construction method based on deep learning according to claim 7, is characterized in that, the update point cloud label distribution probability described in step 7, its specific method is: Among them, c _t represents the category probability distribution of the point cloud at time t, Represents a collection of keyframes {K ₀ , K ₁ ,..., K _t }, Z represents a normalization constant, and K _t represents a key frame at time t.