CN115035260A - Indoor mobile robot three-dimensional semantic map construction method - Google Patents

Abstract

The invention discloses a method for constructing a three-dimensional semantic map of an indoor mobile robot. The ORB-SLAM2 algorithm is run through an RGB-D depth camera mounted on the robot to obtain key frame information and pose of the robot, and the RGB map and the depth map of the key frame are reversed through the reverse process. Projection to form a spatial point cloud; filter the spatial point cloud of the key frame and splicing it according to the pose to form a three-dimensional dense point cloud map; use the RGB image of the key frame to use the YOLO V5 network for target recognition to obtain 2D semantic information, and according to The 3D semantic label is obtained by back-projection; the dense point cloud map is segmented by the point cloud segmentation algorithm; the obtained semantic information is fused with the segmentation result of the dense point cloud map to construct a 3D semantic map. By processing the RGB-D information collected by the RGB-D depth camera, the present invention fuses the semantic information with the SLAM result to form a more informative three-dimensional map.

Description

A method for constructing 3D semantic map of indoor mobile robot

technical field

The invention belongs to the technical field of robot semantic map construction, and relates to a robot map construction method, in particular to a construction method of a three-dimensional semantic map of an indoor mobile robot.

Background technique

With the rapid development of intelligent technology, the mobile robot industry has also ushered in vigorous development. Advanced mobile robots are loaded with many sensors with specific functions to complete designated tasks in various scenarios without direct operation by personnel. Among them, for intelligent robots whose application scenarios are indoors, the robot's autonomous exploration method is generally used to complete the map construction task.

At present, the typical semantic construction methods of mobile robots mainly include methods based on visual images and methods based on point cloud processing. For example, in the document "Semantic SLAM Algorithm in Dynamic Environment" by Cheng Zhang et al., they use monocular image sequences to carry out semantic segmentation, Visual odometry and other research work, and finally build a global semantic map; another example is Zhang Zheng et al. In the document "Robust Semantic SLAM Algorithm with Variable Structure", they studied the use of point cloud images to carry out research on the robustness of semantic SLAM.

The above existing technologies have the following problems in constructing a semantic map of mobile robots: first, visual images can only provide RGB color information, and it is difficult to extract three-dimensional spatial information such as point clouds; second, it is not easy to obtain semantic information of the objects represented by point cloud information. , it is difficult to fuse semantics with SLAM mapping and localization results.

SUMMARY OF THE INVENTION

Aiming at the above-mentioned prior art, the technical problem to be solved by the present invention is to provide a method for constructing a three-dimensional semantic map of an indoor mobile robot. By processing the RGB-D information collected by the RGB-D depth camera, the semantic information and the SLAM result are fused. , to form a more informative 3D map.

In order to solve the above technical problems, a method for constructing a three-dimensional semantic map of an indoor mobile robot of the present invention includes the following steps:

Step 1: Run the ORB-SLAM2 algorithm through the RGB-D depth camera carried by the robot to obtain the key frame information and corresponding pose of the robot, and back-project the RGB image and depth image corresponding to each key frame to form a single-frame spatial point cloud;

Step 2: Filtering the spatial point cloud corresponding to the key frame, and splicing according to the pose corresponding to the key frame to form a three-dimensional dense point cloud map;

Step 3: Use the YOLO V5 network to identify the RGB image corresponding to the key frame, obtain the 2D semantic information in the key frame, and obtain the 3D semantic label according to the back projection;

Step 4: Segment the dense point cloud map through the point cloud segmentation algorithm;

Step 5: Integrate the obtained semantic information with the segmentation result of the dense point cloud map to construct a 3D semantic map.

Further, step 1 includes:

The processing of key frames is added to the Tracking thread in the ORB-SLAM2 system to obtain the spatial point cloud data corresponding to the key frames. Specifically, the following operations are performed on all the pixels in the key frames to obtain the key frames in the camera coordinate system. Point cloud data: Let the coordinate of a pixel point P' in the image be [u, v] ^T , the camera's internal parameter matrix is K, and the depth data corresponding to the pixel point is d, then convert the pixel point to the camera coordinate system. The three-dimensional point P _c (X', Y', Z') is:

After obtaining the spatial coordinates of the three-dimensional point P _c , then obtain the RGB information of the pixel point P' as the RGB information of P _c , and perform the above operations on all the pixels in the key frame. Through the point cloud constructor of the PCL library, the described The point cloud data of the key frame in the camera coordinate system;

Then convert the point cloud data of the key frame in the camera coordinate system to the point cloud data in the world coordinate system:

Let P _c be converted to a point in the world coordinate system as P _w (X, Y, Z), and P _w (X, Y, Z) satisfies:

where T _cw is the camera pose.

Further, filtering the spatial point cloud corresponding to the key frame in step 2 includes:

Use the de-outlier filter in the PCL point cloud library to remove outliers in the point cloud data, and calculate the average distance from each point in the spatial point cloud to all adjacent points. The average distance is at the outlier distance threshold d. Points outside _pt are considered outliers and removed from the data. Then, the point cloud data is down-sampled by the method of voxel grid filtering.

Further, the outlier distance threshold d _pt is specifically:

Assuming that the distances of all points in the spatial point cloud obey a normal distribution (μ,σ ² ), and the threshold parameter for judging whether it is an outlier is H _out , the outlier distance threshold d _pt is:

d _pt = μ + H _out ·σ.

Further, in step 2, splicing according to the pose corresponding to the key frame to form a three-dimensional dense point cloud map includes:

The LoopClosing thread in the ORB-SLAM2 system optimizes the detected key frame pose in the closed loop, extracts the key frame where the camera pose changes after the closed loop optimization, and then deletes the point cloud corresponding to the key frame in the world coordinate system The corresponding point cloud data is regenerated according to the closed-loop optimized camera pose, and then projected into the world coordinate system to construct a three-dimensional dense point cloud map.

Further, step 3 includes:

Step 3.1: Build a dataset based on the YOLO V5 network model

Take a picture of the target object to get a data set, use the labeling program Labelimg to label the collected data set, and label it, and select the training set and test set from the captured pictures; select the pictures containing the target object from the COCO data set as the training set. and test set; perform data configuration and model training parameter settings for the YOLO V5 network, the data configuration includes the number of object categories and object category names, use the training set to train the YOLO V5 network model parameters, and use the test set to verify the network model;

Step 3.2: Extract 2D object semantics and build 3D semantic labels

Make the YOLO V5 network call the RGB-D depth camera to collect RGB image information, and obtain the 2D recognition frame in the RGB image in real time as the 2D target semantic information;

Add an interface to the Tracking thread of ORB-SLAM2 to read the key frames generated in real time, and use the trained YOLO V5 network model to perform target detection on each newly generated key frame to obtain the pixel coordinates of the center point of the target object and the length of the bounding box. and width, and then generate spatial point cloud data according to the RGB map and depth map corresponding to the key frame, change the RGB information of all pixels in the 2D bounding box of the target object to the set color information. A single frame of point cloud data has 3D semantic labels, and then the obtained key frame point cloud data combined with the result of pose estimation are back-projected into the world coordinate system to complete point cloud splicing, and a global dense point cloud map with 3D semantic labels is obtained. That is, add color information to the point cloud of the target object in the point cloud map.

Further, in step 4, the dense point cloud map is segmented by the point cloud segmentation algorithm, including:

Step 4.1: Use the hypervoxel clustering segmentation method to segment the dense point cloud map, and convert the point cloud into a surface structure. The segmentation result is represented by an adjacency graph G={v,ε}, where v _i ∈ v is obtained by segmentation Surface block, ε represents connecting adjacent surface blocks (v _i , v _j ), each surface block contains a centroid c _i and a normal vector _ni ;

Step 4.2: After obtaining the surface block, use RANSAC to process the surface block to generate a candidate plane PC={pc ₁ ,pc ₂ ,...,pc _m }, and then calculate the distance _d (c _{i ,} pc _m ), set the threshold δ, and obtain all the surface blocks whose distance to the plane pc _m is within δ, named Π={vi _∈V |d(ci ,pc _m )<δ _} , defined as:

Among them, η represents the constraints to distinguish the foreground object from the background, D(p _m ) represents the weight of the point cloud plane, and the minimized energy P ^* of the graph segmentation problem is:

Among them, E(P) is the fitted energy, which satisfies:

Among them, V and E are the set of vertices and edges in the graph, respectively, corresponding to the adjacency graph G={v,ε} of cluster segmentation.

Further, step 5 includes:

According to the 2D recognition frame, the bounding box of the point cloud of the target object is obtained, wherein the width value b of the bounding box is the maximum depth difference corresponding to all the pixels in the 2D recognition frame. It is assumed that all the pixels in the 2D recognition frame are the set P={ P ₁ , P ₂ , P ₃ ......, P _n }, then

b=max(Depth(P _i )-Depth(P _j ))

Among them, P _i , P _j ∈ P, Depth(P _i ) is the depth value corresponding to the pixel point P _i , and the 2D pixel point is back projected into the world coordinate system, and the length and width of the 2D recognition frame are mapped to the world coordinate system. to get the length and height of the bounding box;

Then, the results obtained by point cloud segmentation are detected. Only when all point clouds in a cluster are located in the bounding box, replace the random color attribute label of the cluster with the 3D semantic label obtained in step 3 to obtain a semantic map. .

Further, the RGB-D depth camera adopts the Kinect V2 camera.

Further, the YOLO V5 network adopts the YOLOv5s network.

Beneficial effects of the present invention: the present invention obtains the RGB-D information of the indoor environment through a robot equipped with an RGB-D depth camera (such as a Kinect camera), and based on the key frames generated by the ORB-SLAM2 system, through the trained YOLO v5s network model, After completing the target recognition, the semantic information is converted into 3D semantic labels. And according to the RGB map corresponding to the key frame and the depth map combined with the pose information of the key frame, a three-dimensional dense point cloud map is generated. On this basis, the point cloud map is segmented, and the semantic labels are fused with the dense point cloud map to construct rich information. 3D semantic map.

Description of drawings

Fig. 1 is the flow chart of the present invention;

Fig. 2 is the flow chart of dense point cloud construction;

Figure 3 is a dense point cloud map;

Fig. 4 is a part of the data set samples collected;

Figure 5 is the calibration of the data sample;

Figure 6 shows the real-time detection results of YOLO v5s;

Fig. 7 is the flow chart of generating point cloud label;

Figure 8 is a point cloud label map in a real scene;

Fig. 9 is the supervoxel clustering segmentation result;

Figure 10 is the result of point cloud segmentation;

Figure 11 shows the fused semantic map.

Detailed ways

The present invention will be further described below with reference to the accompanying drawings and embodiments of the description.

The flow chart of the present invention is shown in Figure 1. The present invention selects the Kinect V2 camera as the visual sensor, obtains the key frames in the camera movement process based on the ORB-SLAM 2 algorithm, and uses the process of feature point matching and back-end optimization in the key frames to obtain After the camera pose, the corresponding RGB image and depth map of each key frame are back-projected to form a spatial point cloud. After filtering, the camera pose information is combined to form a three-dimensional dense point cloud map. At the same time, YOLO V5 is used. The network model performs target recognition on the RGB map corresponding to the key frame, obtains the semantic information of the target, and projects it to the spatial point cloud to realize the construction of 3D semantic label; then uses the point cloud segmentation algorithm to segment the dense point cloud map, and finally divides The obtained semantic information is fused with the segmentation results of the dense point cloud map to construct a 3D semantic map.

1. Improve ORB-SLAM2 to build 3D dense point cloud

The present invention achieves the goal of obtaining dense point clouds by improving some threads in ORB-SLAM2. The specific process is shown in Figure 2, and the main steps are as follows:

Step 1: Back-projection of 2D pixel points to 3D point cloud data

Modified the Tracking thread in the ORB-SLAM2 system, increased the processing of key frames, and obtained the 3D point cloud data corresponding to the key frames. The principle of this process is to back-project the two-dimensional pixel coordinates and depth data of the environment to the world coordinate system through the camera's internal and external parameters. Let the coordinates of a pixel point P' in the image be [u, v] ^T , and the camera's internal parameter matrix is K, the depth data corresponding to the pixel point is d; then the pixel point is converted into a three-dimensional point P _c (X', Y', Z') in the camera coordinate system, which can be expressed as follows:

After obtaining the spatial coordinates of the three-dimensional point P _c , then obtain the RGB information of the pixel point P' as the RGB information of P _c , and perform the above operations on all the pixels in the key frame. Through the point cloud constructor of the PCL library, the described The point cloud data of the key frame in the camera coordinate system. After obtaining the point cloud data corresponding to the key frame, convert it to a point cloud in the world coordinate system. Let P _c be converted to a point in the world coordinate system as P _w (X, Y, Z), and the process can use the following formula represents, where T _cw is the camera pose:

Step2: Dense point cloud data filtering

The invention uses the outlier filter in the PCL point cloud library to remove abnormal noise points in the point cloud data, and the point cloud obtained after the outlier filter is relatively smooth, which is suitable for subsequent processing of point cloud clustering. Then, the point cloud data is down-sampled by the method of voxel grid filtering. When the present invention filters out outliers, for each point, the average distance from it to all adjacent points is calculated, and the obtained result should obey the Gaussian distribution, and its shape is determined by the Gaussian distribution parameters (μ, σ ² ). Points with distances outside the standard range are considered outliers and removed from the data.

Assuming that the distances of all points in the point cloud obey a normal distribution (μ,σ ² ), and the threshold parameter for judging whether it is an outlier is H _out , the outlier distance threshold d _pt can be expressed as:

d _pt = μ+H _out ·σ (3)

That is, points whose average distance is outside the interval (0,d _pt ) will be considered as outliers and filtered out.

The present invention conducts experiments. Based on the hardware platform (Intel Core i5-10400F CPU, NVIDIA GTX1650), the resolution parameter of the voxel grid filtering is adjusted to 0.01, that is, when the size of the voxel grid is 1cm*1cm*1cm, it can be obtained the best computing effect.

Step3: point cloud stitching

In the present invention, the global dense point cloud data obtained directly according to step1 depends on the camera pose T _cw of each key frame, but the LoopClosing thread in the ORB-SLAM2 system will adjust the position of the key frame in the entire closed loop according to the detected closed loop. At this time, the pose estimation results of some key frames will change.

In order to avoid ghosting in the constructed dense point cloud map, the present invention extracts the key frame where the camera pose changes after the closed-loop optimization, and then deletes the point cloud data corresponding to the key frame in the world coordinate system. The corresponding point cloud data is regenerated from the pose, projected into the world coordinate system, and the point cloud stitching after loop closure detection is completed. Finally, the 3D dense point cloud map constructed based on the ORB-SLAM2 system is shown in Figure 3.

2. 3D semantic label construction based on YOLO v5s network model

The present invention needs to perform target recognition on the key frames generated by the Tracking thread in the ORB SLAM2 system, and then map the obtained 2D semantic information into the spatial point cloud data to complete the construction of the 3D semantic label. The specific steps are as follows:

Step1: Establish a dataset based on the YOLO v5s network model

This patent uses a camera to photograph common objects in the laboratory, mainly including common items such as computer equipment and books. Part of the collected images are shown in Figure 4. Use the labeling program Labelimg to mark and label the collected data set, as shown in Figure 5. For example, the process of labeling "display" is marked with "tv" label. 500 images were selected as the training set and 200 as the test set. In addition, 1500 pictures containing objects in indoor scenes are selected from COCO as the training set and 300 pictures are used as the test set. In order to use with the COCO dataset, the annotation files are stored in json file format. In the present invention, there are 35 commonly used objects in the indoor environment.

Before training, you need to modify the YOLO v5s data and the configuration file of the model, change the number of object categories nc of the two files to 35, and modify the name of the object category included in the category list names. There are 2000 images in the training set and 500 images in the test set. After the data set is prepared, the parameters of the YOLO v5s network model are trained. The hyperparameter configuration of the training model is shown in Table 1.

Table 1 Parameter settings of the YOLO v5s network model

Step2: Extract 2D object semantics and construct 3D semantic labels

After the YOLO v5s network training is completed, the Kinect V2 camera is used to collect the corresponding RGB image information in real time, and the real-time detection function can be realized. Semantic label requirements.

The process of mapping 2D semantic information to 3D semantic labels is shown in Figure 7. The first step is to add an interface to the Tracking thread of ORB-SLAM2. Based on the trained YOLO v5s network, target detection is performed on each newly generated key frame. The pixel coordinates of the center point of the target object and the length and width of the bounding box. Then, when generating spatial point cloud data according to the RGB map and depth map corresponding to the key frame, it is necessary to change the RGB information of all pixels in the 2D bounding box of the target object to the set color information. At this time, the single frame generated according to the key frame The point cloud data has semantic labels. Then, the obtained key frame point cloud data and the result of pose estimation are back-projected into the world coordinate system to complete the point cloud splicing, and the global dense point cloud map with 3D semantic labels is obtained, that is, the point cloud for the target object in the point cloud map. Add color information.

The keyframe generated in ORB-SLAM2 is used as input, and the keyframe image with 2D bounding box and confidence is obtained, and then the corresponding 3D semantic label in the keyframe is obtained through back projection. The result is shown in Figure 8, where the display The point cloud of the keyboard is set to red, the point cloud of the keyboard is set to green, and the point cloud of the mouse is set to blue.

3. Dense point cloud segmentation

The present invention performs point cloud segmentation on the constructed dense point cloud map, so as to obtain a scene understandable by the robot, and at the same time obtains an accurate target point cloud according to the segmentation result of the dense point cloud map, which is convenient for subsequent combination with 3D semantic labels. First, the original dense point cloud map is segmented using a segmentation method based on super-voxel clustering, and the irregular point cloud is converted into a surface structure according to the similarity of points. The patches formed after super-voxel segmentation have A centroid and a normal vector, point cloud segmentation can be defined as a graph segmentation problem, and then the graph cut method is used for further segmentation. The detailed steps are as follows:

Step1: Supervoxel cluster segmentation

Supervoxel segmentation is to use the spatial octree structure of the point cloud, clustering through the class SupervoxelClustering in the PCL library, and add labels to the voxel_centroid_cloud of the octree structure according to the segmentation results, and assign random colors to store in colored_voxel_cloud parameter.

The segmentation principle uses the region growth of k-means clustering to segment the point cloud. The process of superbody clustering is similar to the crystallization process, just like the crystallization of polycrystalline nuclei after the solution is supersaturated. The segmentation method of supervoxel clustering will uniformly distribute the nuclei used for growth in the entire space according to certain rules, and then set the distance between the nuclei and the minimum grains, and the smaller grains will be absorbed by the larger grains. In the process of supervoxel segmentation, the expansion of seed points is determined by the feature distance, which takes into account the feature space calculation of space, color and normal vector. The distance formula in the supervoxel cluster segmentation is as follows:

In the above formula: D _c represents the degree of difference in color, D _n represents the degree of difference in the normal direction, and D _s represents the degree of difference in the distance of the point cloud. In the formula, w _c , _ws , and _wn represent the weight of each variable, respectively. By searching around the crystal nucleus, the voxel with the smallest D value is the most similar, which represents the next crystal nucleus for growth.

The present invention selects the Kinect V2 camera as the sensor, and sets appropriate parameters: voxel_resolution=0.008 (voxel size, resolution of spatial octree), seed_resolution=0.5 (seed resolution). The weights of color, spatial distance and normal vector are adjusted according to the laboratory environment, and the obtained supervoxel clustering segmentation results are shown in Figure 9.

Step2: Point cloud segmentation based on geometric information

The _supervoxel segmentation result obtained in the previous step is represented by an adjacency graph G={v,ε}, where vi ∈ v is the surface block obtained by segmentation, and ε represents the connection between adjacent surface blocks ( _{vi , v j )} . After supervoxel segmentation, each surface block has a centroid c _i and a normal vector _ni . At this time, scene segmentation can be regarded as a graph segmentation problem.

Suppose there are K actual planes {s ₁ ,s ₂ ,...,s _k } in a point cloud map, and these planes have been divided into surface patches. A variable {b _i } ₁ ^N is defined here, b _i ∈[0,K], b _i =K indicates that the surface block belongs to the plane _sk , and N represents that the actual plane _sk is divided into N surface blocks, as shown in the figure 9 The desktop is divided into two clusters. If the planes of all objects in the point cloud map can be extracted and the surface blocks can be assigned to them, a more accurate point cloud segmentation result can be obtained.

The specific operation After obtaining the surface block, use RANSAC to process the surface block to generate a candidate plane PC={pc ₁ ,pc ₂ ,...,pc _m }, and calculate d(ci ,pc _m ), that is _{, the centroid of the surface block c i to} the plane pc _m distance. By adding a threshold δ, all the surface blocks whose distance to the plane pc _m is within δ can be obtained, named ∏={vi _∈V |d(ci ,pc _m )<δ _} . Define the following formula:

In the above formula: η represents the constraint to distinguish the foreground object from the background, and D(p _m ) represents the weight of the point cloud plane. In the experiment, η=40 and δ=4 (cm) were set. The minimization energy P ^* of the graph segmentation problem can be expressed as:

In the above formula, E(P) is the fitted energy, as shown in the following formula:

Among them, V and E are the set of vertices and edges in the graph, respectively, corresponding to the adjacency graph G={v,ε} of cluster segmentation.

Based on the minimum cut theory of point cloud, the edge when the energy is minimized is cut off, that is, the segmentation result of the point cloud can be obtained by merging the adjacent surface blocks according to the constraints. The clustering and merging of objects, the segmentation result is shown in Figure 10. Among them, the segmentation effect of target objects such as monitors, books, cups, bottles, etc. is very good, and the merger effect of background objects such as tables, lockers, ground and other planes is also very good. meet the requirements of the present invention.

4. Integrate semantic information to build a semantic map

After completing the segmentation of the dense point cloud, the present invention combines the segmentation result with the 3D semantic label, optimizes the segmentation result through the semantic information, and can segment the target that cannot be extracted in the point cloud segmentation; Each target adds semantic information to build a more accurate semantic map. After point cloud segmentation, each cluster has an attribute label. After replacing the attribute label of the cluster with the semantic label of the specific target, the specific object in the point cloud has semantic information, thereby constructing a semantic map. The specific implementation steps are as follows:

Step1: Storage of Semantic Labels

In the present invention, after the dense point cloud map is segmented, the point clouds of different colors represent the point cloud set and the spatial sampling area in each supervoxel, and the color is the attribute label of the cluster. However, the attribute label at this time only distinguishes each clustering result, which itself is a random label without any meaning, and this label is also the attribute label of each cluster in the final point cloud segmentation result.

In the second part, the present invention uses YOLO v5s to perform target recognition on the key frames generated in ORB-SLAM2, and according to the 3D semantic labels obtained from the recognition results, add color labels with semantic information to the results of point cloud segmentation, and complete Storage of semantic labels for objects.

However, at this time, all the pixels in the 2D target recognition bounding box are directly mapped to the 3D semantic label formed by combining the depth data of the Kinect camera. In the complex environment of the laboratory, the bounding box of target recognition is not accurate enough, and it is easy to add wrong semantic information to the surrounding point cloud of the target object. As shown in Figure 8, there is a wrong semantic label added to the surrounding points of the target object. Situation in the cloud.

Step2: Fusion of semantic labels and point cloud bounding boxes

In order to obtain a more accurate semantic map, the present invention obtains the bounding box of the point cloud of the target object according to the 2D recognition frame, wherein the width value b of the bounding box is the maximum depth difference corresponding to all pixels in the 2D recognition frame. Assuming that all the pixels in the 2D recognition frame are set P={P ₁ , P ₂ , P ₃ ......, P _n }, then

b=max(Depth(P _i )-Depth(P _j )) (8)

Among them, P _i , P _j ∈P, Depth(P _i ) is the depth value corresponding to the pixel point P _i . The length and height of the bounding box are obtained by mapping the length and width of the 2D recognition frame. The principle is that the 2D pixels are back-projected into the world coordinate system, which will not be repeated here.

Next, the present invention will detect the result obtained by point cloud segmentation, and only when all point clouds in a certain cluster are located in the bounding box, will specific semantic information be added, that is, a specific color will be added to it attributes to obtain a more precise semantic map. As shown in Figure 11 and Figure 10, through the fusion of the bounding box obtained by the semantic label and the point cloud segmentation result, the object TV can be segmented more completely, and the clustering of the actual object edge point cloud has also successfully added correct semantic information ; Compared with adding semantic information to the point cloud directly through the bounding box obtained by 2D target recognition in Figure 8, this method can also effectively avoid adding semantic information to the point cloud around the target by mistake. At the same time, combining with semantic information can also improve the effect of point cloud segmentation. As shown in Figure 11, the added semantic information changes the color attribute and then re-clusters, which can segment small objects such as keyboard and mouse on the desktop. According to the above two steps, the three-dimensional semantic map as shown in FIG. 11 can be formed.

Claims (10)

1. A three-dimensional semantic map construction method for an indoor mobile robot is characterized by comprising the following steps:

step 1: running an ORB-SLAM2 algorithm through an RGB-D depth camera carried by the robot to obtain key frame information and corresponding poses of the robot, and forming a single-frame space point cloud by back projection of an RGB image and a depth image corresponding to each key frame;

step 2: filtering the spatial point cloud corresponding to the key frame, and splicing according to the pose corresponding to the key frame to form a three-dimensional dense point cloud map;

and step 3: performing target identification on the RGB image corresponding to the key frame by using a YOLO V5 network to obtain 2D semantic information in the key frame, and obtaining a 3D semantic label according to back projection;

and 4, step 4: partitioning the dense point cloud map by a point cloud partitioning algorithm;

and 5: and fusing the obtained semantic information with the segmentation result of the dense point cloud map to construct and obtain the 3D semantic map.

2. The indoor mobile robot three-dimensional semantic map construction method according to claim 1, characterized in that: the step1 comprises the following steps:

adding processing on a key frame in a Tracking thread in an ORB-SLAM2 system to obtain spatial point cloud data corresponding to the key frame, specifically: performing the following operations on all pixel points in the key frame to obtain point cloud data of the key frame under a camera coordinate system: let the coordinate of a certain pixel point P' in the image be [ u, v] ^T If the internal reference matrix of the camera is K and the depth data corresponding to the pixel point is d, converting the pixel point into a three-dimensional point P under a camera coordinate system _c (X ', Y ', Z ') is:

obtain a three-dimensional point P _c After the space coordinate, the RGB information of the pixel point P' is acquired as P _c The RGB information of the key frame is operated as above, and point cloud data of the key frame under a camera coordinate system is obtained through a point cloud construction function of a PCL library;

and then converting point cloud data of key frames under a camera coordinate system into point cloud data under a world coordinate system:

let P _c Converted into a point P in the world coordinate system _w (X,Y,Z)，P _w (X, Y, Z) satisfies:

wherein, T _cw Is the camera pose.

3. The indoor mobile robot three-dimensional semantic map construction method according to claim 1, characterized in that: step2, the filtering processing of the spatial point cloud corresponding to the key frame comprises:

removing outliers in the point cloud data by using an outlier removing filter in the PCL point cloud base, calculating the average distance from each point in the spatial point cloud to all adjacent points, wherein the average distance is within an outlier distance threshold d _pt The outliers, are identified as outliers and removed from the data. And then, point cloud data is down-sampled by a voxel grid filtering method.

4. The indoor mobile robot three-dimensional semantic map construction method according to claim 3, characterized in that: the outlier distance threshold d _pt The method specifically comprises the following steps:

the distances of all points in the spatial point cloud are arranged to obey normal distribution (mu, sigma) ² ) Judging whether the threshold parameter of the outlier is H _out Then the outlier distance threshold d _pt Comprises the following steps:

d _pt ＝μ+H _out ·σ。

5. the indoor mobile robot three-dimensional semantic map construction method according to claim 1, characterized in that: step2, the step of splicing the poses corresponding to the key frames to form the three-dimensional dense point cloud map comprises the following steps:

the Loopcolising thread in the ORB-SLAM2 system optimizes the detected key frame pose in the closed loop, extracts the key frame with the changed camera pose after the closed loop optimization, deletes the point cloud data corresponding to the key frame in the world coordinate system, regenerates the corresponding point cloud data according to the camera pose after the closed loop optimization, and projects the point cloud data into the world coordinate system to construct the three-dimensional dense point cloud map.

6. The indoor mobile robot three-dimensional semantic map construction method according to claim 1, characterized in that: the step3 comprises the following steps:

step 3.1: establishing data set based on YOLO V5 network model

Shooting a picture of a target object to obtain a data set, labeling the collected data set by using a labeling program Labelimg, labeling, and respectively selecting a training set and a test set from the shot picture; selecting pictures containing target objects from the COCO data set as a training set and a test set respectively; carrying out data configuration and model training parameter setting on a YOLO V5 network, wherein the data configuration comprises object class number and object class name, training parameters of a YOLO V5 network model by using a training set, and verifying the network model by using a test set;

step 3.2: extracting 2D object semantics and constructing 3D semantic tags

Enabling a YOLO V5 network to call an RGB-D depth camera to acquire RGB image information, and acquiring a 2D recognition frame in the RGB image in real time to serve as 2D target semantic information;

adding an interface in a Tracking thread of ORB-SLAM2 to read key frames generated in real time, performing target detection on each newly generated key frame by using a trained YOLO V5 network model to obtain the pixel coordinates of the center point of a target object and the length and width of a boundary frame, then changing the RGB information of all pixel points in a 2D boundary frame of the target object into set color information when generating space point cloud data according to an RGB image and a depth image corresponding to the key frame, wherein the single-frame point cloud data generated according to the key frame has a 3D semantic label, then reversely projecting the obtained key frame point cloud data and a result of pose estimation to a world coordinate system to complete point cloud splicing, and obtaining a global dense map with the 3D semantic label, namely adding color information to the point cloud of the target object in the point cloud map.

7. The indoor mobile robot three-dimensional semantic map construction method according to claim 1, characterized in that: 4, the step of partitioning the dense point cloud map by using a point cloud partitioning algorithm comprises the following steps:

step 4.1: and (3) partitioning the dense point cloud map by adopting a hyper-voxel clustering partitioning method, converting the point cloud into a surface structure, and representing a partitioning result by adopting an adjacency graph G ═ v, epsilon, wherein v is represented by _i Epsilon.v is a curved surface block obtained by dividing, epsilon represents a curved surface block (v) connecting adjacent curved surface blocks _i ,v _j ) Each of which isThe curved surface block comprises a center of mass c _i And a normal vector n _i ；

Step 4.2: after the curved surface block is obtained, the curved surface block is processed by RANSAC to generate a candidate plane PC ═ PC ₁ ,pc ₂ ,…,pc _m Then calculate the centroid c of the surface patch _i To the plane pc _m Distance d (c) of _i ,pc _m ) Setting a threshold delta to obtain all arrival planes pc _m The surface patch with distance within delta is named pi ═ v _i ∈V|d(c _i ,pc _m ) < delta >, define:

where η represents the constraint that distinguishes foreground objects from the background, D (pc) _m ) Minimizing energy P representing weight of point cloud plane, graph partitioning problem ^* Comprises the following steps:

P ^* ＝arg min E(P),

wherein E (P) is the energy of the fitting, and satisfies:

wherein V and E are the set of the top point and the edge in the graph respectively, and the adjacency graph G corresponding to the cluster segmentation is { V, epsilon }.

8. The indoor mobile robot three-dimensional semantic map construction method according to claim 1, characterized in that: the step 5 comprises the following steps:

obtaining a bounding box of the point cloud of the target object according to the 2D identification frame, wherein the width value b of the bounding box is the maximum depth difference corresponding to all pixel points in the 2D identification frame, and assuming that all pixel points in the 2D identification frame are a set P ═ P ₁ ，P ₂ ，P ₃ ......，P _n }，Then

b＝max(Depth(P _i )-Depth(P _j ))

Wherein, P _i ,P _j ∈P,Depth(P _i ) Is a pixel point P _i The corresponding depth value is back projected into a world coordinate system by utilizing the 2D pixel points, and the length and the width of the 2D identification frame are mapped into the world coordinate system to obtain the length and the height of the bounding box;

and detecting the result obtained by point cloud segmentation, and replacing the random color attribute labels of the clusters with the 3D semantic labels obtained in the step (3) to obtain a semantic map only when all point clouds in a certain cluster are located in the bounding box.

9. The method for constructing the three-dimensional semantic map of the indoor mobile robot according to any one of claims 1 to 8, wherein: the RGB-D depth camera employs a Kinect V2 camera.

10. The method for constructing the three-dimensional semantic map of the indoor mobile robot according to any one of claims 1 to 8, wherein: the YOLO V5 network adopts a YOLOv5s network.

Images