CN118129729A

CN118129729A - Mapping method, mapping device, mobile robot and computer readable storage medium

Info

Publication number: CN118129729A
Application number: CN202410227267.8A
Authority: CN
Inventors: 卢涛; 陈依然; 凌勇; 徐成禄
Original assignee: Yunjing Intelligent Innovation Shenzhen Co ltd; Yunjing Intelligent Shenzhen Co Ltd
Current assignee: Yunjing Intelligent Innovation Shenzhen Co ltd; Yunjing Intelligent Shenzhen Co Ltd
Priority date: 2024-02-29
Filing date: 2024-02-29
Publication date: 2024-06-04

Abstract

The embodiment of the invention provides a mapping method, a mapping device, a mobile robot and a computer readable storage medium. The mapping method comprises the following steps: acquiring semantic segmentation information and first point cloud data corresponding to visual data, wherein the visual sensor acquires the visual data aiming at a target area, and the ranging sensor acquires the first point cloud data aiming at the target area; fusing the first point cloud data and semantic segmentation information to obtain second point cloud data with semantic information; and constructing a target map with semantic information based on the second point cloud data. The scheme can accurately construct the target map with semantic information.

Description

Mapping method, mapping device, mobile robot and computer readable storage medium

Technical Field

The present invention relates to the field of mobile robots, and more particularly, to a mapping method, a mapping apparatus, a mobile robot, and a computer-readable storage medium.

Background

With the rapid development of artificial intelligence (ARTIFICIAL INTELLIGENCE, AI) technology, various mobile robots are increasingly entering the lives of people, and great convenience is brought to the lives of people. Common mobile robots include sweeping robots, transfer robots, and the like. After the mobile robot enters a strange environment, a map can be created by combining technologies such as instant positioning and map construction (Simultaneous Localizationand Mapping, SLAM) and the like so as to plan a path and execute tasks based on the map later.

In the related art, radar point cloud data acquired by a radar sensor arranged on a mobile robot is generally used for establishing a radar map, or a three-dimensional (3 d) point cloud with semantics is obtained by using a depth camera and a semantic segmentation mode and projected to a two-dimensional (2 d) map, so that a semantic map is obtained. The radar map itself is highly accurate but lacks semantic information. While the semantic map includes semantic information, the geometric error itself is large. At present, a map construction method with higher precision and semantic information is lacking.

Disclosure of Invention

The present invention has been made in view of the above-described problems. According to an aspect of the present invention, there is provided a mapping method applied to a mobile robot including a vision sensor and a ranging sensor; comprising the following steps: acquiring semantic segmentation information and first point cloud data corresponding to visual data, wherein the visual sensor acquires the visual data aiming at a target area, and the ranging sensor acquires the first point cloud data aiming at the target area; fusing the first point cloud data and semantic segmentation information to obtain second point cloud data with semantic information; and constructing a target map with semantic information based on the second point cloud data.

Illustratively, the semantic segmentation information includes semantic information that corresponds one-to-one to pixel points in the visual data; fusing the first point cloud data and the semantic segmentation information to obtain second point cloud data with semantic information, comprising: projecting the ranging points in the first point cloud data to a pixel coordinate system corresponding to the visual data to obtain projection points corresponding to the ranging points in the first point cloud data one by one; and determining target semantic information of at least part of ranging points in the first point cloud data at least based on semantic information corresponding to the pixel points at the coordinates of each projection point so as to obtain second point cloud data.

Illustratively, determining target semantic information of at least part of the ranging points in the first point cloud data based at least on semantic information corresponding to the pixel point at the coordinates of each projection point to obtain second point cloud data includes: determining target semantic information of an effective ranging point set based on at least semantic information corresponding to pixel points at coordinates of each projection point, wherein the effective ranging point set comprises ranging points meeting validity requirements in first point cloud data; and setting target semantic information corresponding to the ranging point with the target semantic information error in the effective ranging point set as unknown, or deleting the ranging point with the target semantic information error in the effective ranging point set to obtain second point cloud data, wherein the corresponding second point cloud data comprises the ranging points with the known target semantic information in the effective ranging point set, or the second point cloud data comprises the rest ranging points in the effective ranging point set.

Illustratively, before determining the target semantic information for the set of valid ranging points based at least on the semantic information corresponding to the pixel point at the coordinates where each projected point is located, the method further comprises: deleting ranging points corresponding to projection points which are not in the visual field range of the visual sensor in the first point cloud data to obtain an effective ranging point set; wherein the validity requirements include: the ranging points correspond to projected points that are within the field of view of the vision sensor.

Illustratively, determining target semantic information for the set of valid ranging points based at least on semantic information corresponding to a pixel point at the coordinates where each projected point is located, includes: for each ranging point in the effective ranging point set, judging whether semantic information corresponding to each pixel point in an associated area is the same, wherein the associated area is an area with a preset size taking a projection point corresponding to the ranging point as a center; when the semantic information corresponding to each pixel point in the associated area is the same, determining the semantic information corresponding to any pixel point in the associated area as the target semantic information of the ranging point; and/or determining that the target semantic information of the ranging point is wrong when the semantic information corresponding to each pixel point in the associated area is not identical.

Illustratively, setting the target semantic information corresponding to the ranging point with the target semantic information error in the valid ranging point set as unknown, or deleting the ranging point with the target semantic information error in the valid ranging point set, includes: for each ranging point in the effective ranging point set, judging whether the distance between the ranging point and the mobile robot meets the distance requirement corresponding to the target semantic information corresponding to the ranging point; when the distance between the ranging point and the mobile robot does not meet the distance requirement, determining that the target semantic information of the ranging point is wrong, setting the target semantic information corresponding to the ranging point as unknown, or deleting the ranging point.

Illustratively, setting the target semantic information corresponding to the ranging point with the target semantic information error in the valid ranging point set as unknown, or deleting the ranging point with the target semantic information error in the valid ranging point set, includes: traversing the ranging points in the effective ranging point set, and dividing the ranging points adjacent to the index and corresponding to the target semantic information into at least one ranging point group for any target semantic information; for each ranging point group in at least one ranging point group, calculating a mean value of distances between each ranging point in the ranging point group and the mobile robot to determine the distance between the ranging point group and the mobile robot; setting target semantic information corresponding to a specific ranging point group in at least one ranging point group as unknown, or deleting the specific ranging point group in the at least one ranging point group, wherein the specific ranging point group is other ranging point groups except for the ranging point group with the minimum distance with the mobile robot; the distance between any two adjacent ranging points in each ranging point group is smaller than or equal to a preset distance threshold, the distance between the ranging points corresponding to any two ranging point groups is larger than the preset distance threshold, and the ranging points with the target semantic information errors comprise ranging points in a specific ranging point group.

Illustratively, constructing the target map with semantic information based on the second point cloud data includes: storing semantic information of each ranging point in the second point cloud data into a corresponding map point in the target map, wherein the semantic information of each ranging point is a group of semantic information; for each map point in the target map, when the number of semantic information stored in the map point reaches a preset number, determining that the semantic information with the largest repeated occurrence among the semantic information stored in the map point is the semantic information corresponding to the map point.

According to a second aspect of the present invention, there is also provided a mapping apparatus applied to a mobile robot including a vision sensor and a ranging sensor; the device comprises: the acquisition module is used for acquiring semantic segmentation information and first point cloud data corresponding to the visual data, wherein the visual sensor acquires the visual data aiming at the target area, and the ranging sensor acquires the first point cloud data aiming at the target area; the fusion module is used for fusing the first point cloud data and the semantic segmentation information to obtain second point cloud data with semantic information; and the construction module is used for constructing a target map with semantic information based on the second point cloud data.

According to a third aspect of the present invention, there is also provided a mobile robot including: the computer system comprises a processor and a memory, wherein the memory stores a computer program, and the processor is used for executing the computer program to realize the mapping method.

According to a fourth aspect of the present invention, there is also provided a computer-readable storage medium storing a computer program/instruction which, when executed by a processor, implements the mapping method described above.

According to the technical scheme, the target map with semantic information can be accurately constructed by combining the visual data acquired by the visual sensor and the first point cloud data acquired by the ranging sensor. The target map constructed by the scheme can meet the requirements of high precision and semantic information.

The foregoing description is only an overview of the present invention, and is intended to be implemented in accordance with the teachings of the present invention in order that the same may be more clearly understood and to make the same and other objects, features and advantages of the present invention more readily apparent.

Drawings

The above and other objects, features and advantages of the present invention will become more apparent from the following more particular description of embodiments of the present invention, as illustrated in the accompanying drawings. The accompanying drawings are included to provide a further understanding of embodiments of the invention and are incorporated in and constitute a part of this specification, illustrate the invention and together with the embodiments of the invention, and not constitute a limitation to the invention. In the drawings, like reference numerals generally refer to like parts or steps.

FIG. 1 shows a schematic flow chart of a diagramming method according to one embodiment of the present invention;

FIG. 2 shows a schematic view of a mobile robot according to one embodiment of the invention;

FIG. 3 shows a schematic flow chart of a diagramming method according to a specific embodiment of the present invention;

FIG. 4 shows a schematic block diagram of a mapping apparatus according to one embodiment of the invention;

FIG. 5 shows a schematic block diagram of a mobile robot according to one embodiment of the invention; and

Fig. 6 shows a schematic view of a mobile robot according to an embodiment of the invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. It should be apparent that the described embodiments are only some embodiments of the present invention and not all embodiments of the present invention, and it should be understood that the present invention is not limited by the example embodiments described herein. Based on the embodiments of the invention described in the present application, all other embodiments that a person skilled in the art would have without inventive effort shall fall within the scope of the invention.

According to an aspect of the present invention, there is provided a mapping method applied to a mobile robot including a vision sensor and a ranging sensor.

In the embodiment of the present invention, the mobile robot may be any one of a robot such as a floor sweeping robot, a floor mopping robot, a sweeping and mopping robot, a transfer robot, etc., and the present invention does not limit the kind of the mobile robot.

Alternatively, the visual sensor may be any sensor that may be currently available or developed in the future that may collect visual data of the target area. For example, the visual sensor may be any one of a depth camera, a binocular navigation sensor, a multi-view navigation sensor, a fisheye navigation sensor, and the like. In a specific embodiment, the vision sensor may be a depth camera.

Alternatively, the ranging sensor may be any sensor that can acquire point cloud data of a target area, which is data for representing distance information between the mobile robot and an obstacle, existing or developed in the future. For example, the distance measuring sensor may be a structured light sensor or a radar sensor. In a specific embodiment, the ranging sensor may be a radar sensor.

Fig. 1 shows a schematic flow chart of a mapping method according to an embodiment of the invention. As shown in fig. 1, the mapping method 100 may include step S110, step S120, and step S130.

In step S110, semantic segmentation information and first point cloud data corresponding to the visual data are acquired, wherein the visual data are acquired by the visual sensor for the target area, and the first point cloud data are acquired by the ranging sensor for the target area.

Alternatively, the target area may be set as needed. For example, the target area may be any one of a home space, one room unit of a home space, a partial area of one room unit, a large-sized place, or a partial area of a large-sized place. From another perspective, the target area may refer to a larger area, such as an entire room unit; but also to a partial area in a larger area, for example a room in a room unit or a region in a room. A room unit described herein may include one or more rooms.

In the present exemplary scenario, visual data is acquired by a visual sensor for a target area. The particular type of visual data may be determined based on the type of visual sensor. For example, when the vision sensor is a depth camera, the vision data may be a depth image acquired by the depth camera.

Alternatively, the semantic segmentation information corresponding to the visual data may be obtained by a visual sensor, or may be obtained by a mapping device for performing the mapping method 100. In other words, the step of semantically segmenting the visual data may be performed by a visual sensor or by a mapping device for performing the mapping method 100. The visual data can be two-dimensional visual data or three-dimensional visual data, and correspondingly, the semantic segmentation information corresponding to the visual data can be two-dimensional semantic segmentation information or three-dimensional semantic segmentation information. The depth image is three-dimensional visual data.

Alternatively, the semantic segmentation information corresponding to the visual data may be obtained by a mapping apparatus for performing the mapping method 100. The obtaining of semantic segmentation information corresponding to visual data may specifically include the following steps: acquiring visual data acquired by a visual sensor; and carrying out semantic segmentation on the visual data to obtain semantic segmentation information corresponding to the visual data.

Alternatively, any semantic segmentation model, either existing or developed in the future, may be employed to semantically segment the visual data. For example, the semantic segmentation model may include, but is not limited to, one or more of a U-shaped convolutional neural network (U-net) model, a full convolutional neural network (Fully Convolutional Networks, FCN) model, a DeepLab model, and the like.

Alternatively, the semantic segmentation information may be represented by a semantic segmentation image. Semantic information corresponding to each pixel point can be marked in the semantic segmentation image. For example, different semantic information may be represented using different colors, different thicknesses, different textures, and the like.

In the solution of this example, the first point cloud data is acquired by a ranging sensor for the target area. Optionally, the first point cloud data may be original point cloud data acquired by the ranging sensor for the target area, or may be point cloud data obtained after preprocessing the original point cloud data. The preprocessing operations may include, but are not limited to, denoising, filtering, and the like.

Alternatively, the visual data and the first point cloud data may be acquired by a visual sensor and a ranging sensor for the target area at the same time. Alternatively, the visual data and the first point cloud data may also be acquired by the visual sensor and the ranging sensor for the target area at different times. For example, the visual data may be acquired by a visual sensor for a target area at a first time, and the first point cloud data may be acquired by a ranging sensor for the target area at a second time. Wherein the first time and the second time are different times, and in some embodiments, the difference between the first time and the second time is within a certain threshold, for example, the difference is within 3 seconds. The method for acquiring the visual data and the first point cloud data at different moments can be applied to a scene where the visual data and the first point cloud data are not changed.

In step S120, the first point cloud data and the semantic segmentation information are fused to obtain second point cloud data having semantic information.

After the first point cloud data and the semantic segmentation information are obtained, the first point cloud data and the semantic segmentation information can be fused to obtain second point cloud data. For example, each pixel point in the visual data with semantic segmentation information may be projected into a coordinate system in which the point cloud data is located. For any one ranging point in the first point cloud data, when the ranging point is overlapped with a projection point corresponding to any one pixel point, determining semantic information corresponding to the pixel point as the semantic information of the ranging point. Thus, semantic information of at least part of the ranging points in the first point cloud data can be determined. In this embodiment, the second point cloud data comprises at least part of the ranging points for which semantic information has been determined. For another example, the first point cloud data may be projected into a coordinate system corresponding to the semantic segmentation information to determine the second point cloud data. This scheme is described in detail below and is not repeated. The above manner of fusing the first point cloud data and the semantic segmentation information is merely an example, and is not a limitation of the present invention. Indeed, other ways of fusing the first point cloud data and the semantic segmentation information may also be employed. For example, each pixel point in the first point cloud data and the visual data with semantic segmentation information may be projected into a specific coordinate system (for example, a world coordinate system, a camera coordinate system corresponding to a visual sensor), and the second point cloud data may be determined according to the superposition condition between the projection point corresponding to the first point cloud data and the projection point corresponding to each pixel point.

In step S130, a target map with semantic information is constructed based on the second point cloud data.

After the second point cloud data is obtained, a target map with semantic information can be constructed based on the second point cloud data. For example, the target map may be constructed directly based on the second point cloud data obtained at the present time. Specifically, each ranging point with semantic information in the second point cloud data may be projected onto the target map to obtain the target map with semantic information. For another example, the target map may be constructed by integrating the second point cloud data obtained a plurality of times. In this embodiment, the second point cloud data obtained a plurality of times may be projected onto the target map, respectively. For each map point in the target map, multiple sets of semantic information corresponding to the map point can be integrated to determine the semantic information corresponding to the map point. This scheme is described in detail below and is not repeated.

In the technical scheme, the visual data collected by the visual sensor and the first point cloud data collected by the ranging sensor can be combined to accurately construct the target map with semantic information. The target map constructed by the scheme can meet the requirements of high precision and semantic information.

Illustratively, the semantic segmentation information includes semantic information that corresponds one-to-one to pixel points in the visual data. In step S120, the first point cloud data and the semantic segmentation information are fused to obtain second point cloud data having semantic information, which may include the following steps S121 and S122.

In step S121, the ranging points in the first point cloud data are projected to the pixel coordinate system corresponding to the visual data, so as to obtain projection points corresponding to the ranging points in the first point cloud data one by one.

In step S122, the target semantic information of at least part of the ranging points in the first point cloud data is determined at least based on the semantic information corresponding to the pixel point at the coordinates of each projection point, so as to obtain the second point cloud data.

Optionally, in step S121, projecting the ranging points in the first point cloud data to a pixel coordinate system corresponding to the visual data to obtain projection points corresponding to the ranging points in the first point cloud data one to one, which may include the steps of: projecting the ranging points in the first point cloud data to a camera coordinate system corresponding to the vision sensor to obtain first projection points corresponding to the ranging points in the first point cloud data one by one; the first projection point is projected into a pixel coordinate system to obtain coordinates of a projection point (which may be referred to as a second projection point) of the ranging point in the first point cloud data in the pixel coordinate system. In a specific embodiment, for any ranging point in the first point cloud data, a coordinate transformation matrix (may be referred to as a first coordinate transformation matrix) between a coordinate system where the ranging point is located and a camera coordinate system may be used to determine a first projection point corresponding to the ranging point. Then, a second projection point corresponding to the ranging point may be determined using a coordinate transformation matrix (which may be referred to as a second coordinate transformation matrix) between the camera coordinate system and the pixel coordinate system. The first coordinate transformation matrix and the second coordinate transformation matrix can be determined through a pre-test, and are not described in detail. It will be appreciated that the above-described manner of obtaining projection points in one-to-one correspondence with ranging points in the first point cloud data is merely an example and is not a limitation of the present invention. In some embodiments of the invention, the projected points of the ranging points in the pixel coordinate system may also be obtained by other means. For example, the conversion matrix between the coordinate system in which the ranging point is located and the pixel coordinate system may be directly used to determine the projection point of each ranging point in the first point cloud data in the pixel coordinate system.

As described above, the semantic segmentation information includes semantic information that corresponds one-to-one to pixel points in the visual data. After obtaining the projection points corresponding to the ranging points in the first point cloud data one by one, determining the target semantic information of at least part of the ranging points in the first point cloud data at least based on the semantic information corresponding to the pixel points at the coordinates of each projection point. In some embodiments, for any ranging point in the first point cloud data, it may be determined whether the corresponding semantic information exists in a pixel point of the ranging point at the coordinate where the projection point in the pixel coordinate system is located, and when the corresponding semantic information exists in the pixel point, the semantic information corresponding to the pixel point is directly determined to be the target semantic information corresponding to the ranging point. And executing the judging step on each ranging point in the first point cloud data, thereby obtaining at least part of the ranging points with target semantic information, and determining the at least part of the ranging points as second point cloud data. In other embodiments, after obtaining at least some ranging points with target semantic information, it may be determined whether the target semantic information of each ranging point is correct, and the ranging point with the incorrect target semantic information is deleted or the target semantic information corresponding to the ranging point with the incorrect target semantic information is set as unknown, so as to obtain the second point cloud data. The scheme of deleting the ranging point with the wrong target semantic information or setting the target semantic information corresponding to the ranging point with the wrong target semantic information as unknown is described in detail below, and is not repeated.

According to the technical scheme, the ranging points in the first point cloud data are projected into the pixel coordinate system corresponding to the visual data, so that the semantic information corresponding to the pixel points at the coordinates where the projection points are located can be utilized to accurately determine the target semantic information of at least part of the ranging points in the first point cloud data, and the second point cloud data can be accurately obtained. In a word, the scheme can provide accurate basis for the construction of the target map with the semantic information in the subsequent steps, and is beneficial to improving the accuracy and reliability of the constructed target map with the semantic information.

Alternatively, the validity requirements may be set as desired. For example, for each ranging point, the validity requirements may be: the distance measurement points have corresponding semantic information at the pixel points at the coordinates of the projection points in the pixel coordinate system. In this embodiment, as long as there is corresponding semantic information for the ranging point at the pixel point where the projection point in the pixel coordinate system is located, the ranging point can be considered to satisfy the validity requirement. That is, the ranging point belongs to the set of valid ranging points. For another example, the validity requirements may also be: the ranging points correspond to projected points that are within the field of view of the vision sensor. In this embodiment, a ranging point may be considered to meet the validity requirement when its projected point in the pixel coordinate system is within the field of view of the vision sensor. Specific implementation steps of the scheme are described in detail below and are not repeated.

After determining the target semantic information of the valid ranging point set, the method can continuously judge whether the target semantic information corresponding to each ranging point in the valid ranging point set is correct. For example, whether the target semantic information of the ranging point is correct can be determined according to the distance requirement corresponding to the target semantic information corresponding to the ranging point. For another example, it may be determined whether the projection position of the ranging point in the pixel coordinate system is correct, and when the projection position of the ranging point in the pixel coordinate system is incorrect, the target semantic information of the ranging point is determined to be incorrect. The scheme for determining whether the projection position of the ranging point in the pixel coordinate system is correct is described in detail below, and is not repeated.

After determining the ranging point with the wrong target semantic information in the effective ranging point set, the ranging point can be deleted, or the target semantic information corresponding to the ranging point can be set as unknown. In an alternative embodiment, the target semantic information corresponding to the ranging point with the target semantic information error in the valid ranging point set may be set as unknown, so as to obtain second point cloud data, where the second point cloud data includes the ranging points with known target semantic information in the valid ranging point set. In this embodiment, the ranging points of which the target semantic information is known represent the remaining ranging points in the set of valid ranging points except the ranging points for which the target semantic information is unknown. In another alternative embodiment, the ranging points in the set of valid ranging points that have the wrong target semantic information may be deleted to obtain second point cloud data, which includes the remaining ranging points in the set of valid ranging points. In the embodiment, the distance measurement point with the error target semantic information can be directly deleted, so that the interference of the distance measurement point with the error target semantic information on the subsequent steps can be avoided, the storage space can be saved, and the construction efficiency of the target map can be improved.

According to the technical scheme, the target semantic information of the effective ranging point set is determined, the target semantic information corresponding to the ranging point with the target semantic information error in the effective ranging point set is set to be unknown, or the ranging point with the target semantic information error in the effective ranging point set is deleted, so that the second point cloud data can be determined more accurately, and the accuracy of the constructed target map with the semantic information can be further improved.

It will be appreciated that the vision sensor and the ranging sensor may have different fields of view. For example, and as described above, the semantic segmentation information includes semantic information that corresponds one-to-one to pixels in the visual data, in other words, only pixels that are within the field of view of the visual sensor have semantic information. When the projection point corresponding to the distance measurement point is not in the visual field of the visual sensor, the semantic segmentation information does not comprise the semantic information corresponding to the distance measurement point. Therefore, in the scheme of the present example, the ranging points corresponding to the projection points not within the visual field of the visual sensor in the first point cloud data may be directly deleted, and thus, an effective ranging point set may be obtained.

In the above embodiment of projecting the ranging point in the first point cloud data to the camera coordinate system corresponding to the vision sensor, it may be determined whether the ranging point meets the validity requirement according to the coordinate of the first projection point corresponding to the ranging point in the camera coordinate system. In a specific embodiment, the coordinates of the first projection point corresponding to the ranging point may be expressed as (x, y, z). In this embodiment, it may be determined whether z is greater than 0. When z is less than or equal to 0, the ranging point is determined to not meet the validity requirement. When z > 0, the ranging point is determined to meet the validity requirement.

In the above technical solution, the ranging points corresponding to the projection points not in the visual field of the visual sensor in the first point cloud data may be directly deleted, so as to obtain an effective ranging point set. The method has simple steps, and is beneficial to quickly determining the effective ranging point set, thereby being beneficial to improving the image construction efficiency.

In the solution of this example, the size of the association area may be set according to actual needs. In some embodiments, the association region may be set as a rectangular region centered on the ranging point, having a length of a preset length and a width of a preset width. The preset length and the preset width can be set according to actual needs. For example, the preset length may be in the range of [3,10] pixels and the preset width may be in the range of [2,9] pixels. In a specific embodiment, the preset length and the preset width may each be 3 pixels. In other embodiments, the associated region may be configured as a circular region centered on the ranging point and having a predetermined radius. The setting mode of the preset radius is similar to the setting mode of the preset length and the preset width, and is not repeated.

After the association region is determined, the semantic information corresponding to each pixel point in the association region can be traversed, and whether the semantic information corresponding to each pixel point is the same or not is judged. And when the semantic information corresponding to each pixel point in the associated area is the same, determining that the semantic information of the ranging point is the semantic information corresponding to any pixel point in the associated area. For example, when the semantic information corresponding to each pixel point in the association area is "curtain", the target semantic information corresponding to the ranging point may be determined to be "curtain". When the semantic information corresponding to each pixel point in the associated area is not completely the same, the target semantic information of the ranging point cannot be determined at the moment, so that the target semantic information error of the ranging point can be determined.

In some implementations of the present example, the target semantic information for the ranging point may not be set when the target semantic information for the ranging point is incorrect. It can be appreciated that in a scheme in which the target semantic information of the ranging point is not set, the ranging point does not have the target semantic information. Namely, the target semantic information corresponding to the ranging point is unknown. In other implementations of the present example, for each ranging point in the set of valid ranging points, after determining that the target semantic information for the ranging point is erroneous, the method may further comprise the steps of: the ranging point is deleted from the set of valid ranging points. In this scheme, when the target semantic information of the ranging point is wrong, the ranging point may be deleted from the valid ranging point set. Thus, the ranging point can be prevented from interfering with the subsequent steps.

According to the technical scheme, the target semantic information of each ranging point can be accurately determined by utilizing the semantic information corresponding to each pixel point in the associated area corresponding to each ranging point, and the ranging point with the target semantic information error can be identified, so that accurate basis is provided for determining the second point cloud data in the subsequent step. In short, the scheme is helpful to further improve the accuracy of the constructed target map with semantic information.

In the solution of this example, the distance between the ranging point and the mobile robot is the distance between the ranging point and the ranging sensor of the mobile robot. In some embodiments, the distance between the ranging point and the mobile robot may be determined by coordinates of the ranging point.

Optionally, the distance requirement corresponding to the target semantic information can be set according to actual needs. For example, the distance requirements corresponding to different semantic information may be determined experimentally or empirically in advance. In some embodiments, the distance requirement may be that the distance between the ranging point and the mobile robot is within a preset distance range corresponding to the target semantic information corresponding to the ranging point. In this embodiment, if the distance between the ranging point and the mobile robot is within the distance range corresponding to the target semantic information corresponding to the ranging point, it may be determined that the distance between the ranging point and the mobile robot meets the distance requirement. The preset distance range can be set according to the corresponding target semantic information. In one embodiment, when the target semantic information is "bed" or "sofa", the preset distance range may be [0.5,3.5] meters (m); when the target semantic information is a toilet, the preset distance range can be [0.5,2] m; when the target semantic information is a 'curtain', the preset distance range can be 3,5 m.

It will be appreciated that there may be differences in the optimal distance range for different semantic information. For example, for a larger object, the object may be more fully within the field of view of the vision sensor when the mobile robot is farther from the object, and for a smaller object, the object may be fully and clearly within the field of view of the vision sensor when the mobile robot is closer to the object. In other words, when the object corresponding to the semantic information is large, if the distance between the mobile robot and the object is small, the object corresponding to the semantic information included in the visual data may not be complete enough, and the reliability of the obtained semantic information is poor. When the object corresponding to the semantic information is smaller, if the distance between the mobile robot and the object is larger, the object corresponding to the semantic information included in the visual data may not be clear enough, and at this time, the reliability of the obtained semantic information is poor. In this example, by determining whether the distance between each ranging point and the mobile robot meets the distance requirement corresponding to the target semantic information corresponding to the ranging point, it is helpful to further ensure the accuracy of the determined target semantic information.

Alternatively, the traversal order for the ranging points in the set of valid ranging points may be set as desired. For example, ranging points in the set of valid ranging points may be ordered according to the coordinates of each ranging point in the set of valid ranging points to determine the traversal order. For another example, the traversal order may be determined based on the order in which the ranging sensors collected the ranging points. The above traversal order is merely an example and is not intended to limit the present invention. In some embodiments, the traversal order may also be determined based on the distance between each ranging point in the set of valid ranging points and the mobile robot. For example, the traversal order may be determined in a distance-to-near or near-to-far order from the mobile robot. The method for determining the distance between the ranging point and the mobile robot is described in detail above and will not be repeated.

In the solution of this example, the index of each ranging point in the set of valid ranging points may be determined according to the sequence in which the ranging sensors collect the ranging points. Taking a radar sensor as an example, when the mobile robot collects data by using the radar sensor at a certain position, the radar sensor can rotate 360 degrees and collect data while rotating, and the index can be carried out in the sequence or reverse sequence from 0 degree to 360 degrees. For example, the index of the first ranging point collected by the ranging sensor in the set of valid ranging points may be set to "1", the index of the second ranging point collected by the ranging sensor in the set of valid ranging points may be set to "2" … …, the index of the n-1 st ranging point collected by the ranging sensor in the set of valid ranging points may be set to "n-1", and the index of the n-th ranging point collected by the ranging sensor in the set of valid ranging points may be set to "n". For another example, the index of the nth ranging point collected by the ranging sensor in the valid ranging point set may be set to "1", the index of the (n-1) th ranging point collected by the ranging sensor in the valid ranging point set may be set to "2" … …, the index of the (2) nd ranging point collected by the ranging sensor in the valid ranging point set may be set to "n-1", and the index of the (1) st ranging point collected by the ranging sensor in the valid ranging point set may be set to "n".

Optionally, traversing the ranging points in the valid ranging point set, for any target semantic information, dividing the ranging points adjacent to the index and corresponding to the target semantic information into at least one ranging point group, and may include the following steps: traversing the ranging points in the effective ranging point set, dividing the ranging points with adjacent indexes and same corresponding semantic results into the same target semantic set, wherein different target semantic sets correspond to different target semantic information; for each target semantic set, grouping the ranging points in the target semantic set based on the distance between any two index-adjacent ranging points in the target semantic set to obtain at least one ranging point group corresponding to target semantic information corresponding to the target semantic set. The distance between any two adjacent ranging points can be determined according to the respective coordinates of the two ranging points.

Optionally, based on the distance between any two index adjacent ranging points in the target semantic set, grouping the ranging points in the target semantic set to obtain at least one ranging point group corresponding to the target semantic information corresponding to the target semantic set, which may specifically include the following steps: for any two adjacent ranging points, calculating the distance (which can be called as a first distance) between the two ranging points based on the respective coordinates of the two ranging points; judging whether the first distance is larger than a first distance threshold value; when the first distance is larger than a first distance threshold value, determining that the two ranging points respectively belong to different ranging point groups; and when the first distance is smaller than or equal to the first distance threshold value, determining that the two ranging points belong to the same ranging point group. Alternatively, the first distance threshold may be set according to actual needs. For example, the first distance threshold may be a theoretical value or an empirical value determined experimentally. In some embodiments, different target semantic information may correspond to different first distance thresholds. In this embodiment, different first distance thresholds may be set according to the target semantic information corresponding to the target semantic set. It will be appreciated that when the distance between two ranging points is greater than the distance threshold, it is indicated that the two ranging points are far apart, and the probability that the two ranging points belong to the same object is low, so that the two ranging points can be respectively divided into different ranging point groups.

Optionally, traversing the ranging points in the valid ranging point set, for any target semantic information, dividing the ranging points adjacent to the index and corresponding to the target semantic information into at least one ranging point group, and may include the following steps: traversing the ranging points corresponding to the semantic information of the current target in the effective ranging point set, and grouping the ranging points in the effective ranging point set based on the distance between any two adjacent ranging points to obtain at least one ranging point group corresponding to the semantic information of the current target. The distance between any two adjacent ranging points can be determined according to the respective coordinates of the two ranging points.

Optionally, grouping the ranging points in the valid ranging point set based on the distance between any two neighboring ranging points to obtain at least one ranging point group corresponding to the current target semantic information may specifically include the following steps: for any two adjacent ranging points, calculating the distance (which can be called as a second distance) between the two ranging points based on the respective coordinates of the two ranging points; judging whether the second distance is larger than a second distance threshold value; when the second distance is larger than a second distance threshold value, determining that the two ranging points respectively belong to different ranging point groups; and when the second distance is smaller than or equal to the second distance threshold value, determining that the two ranging points belong to the same ranging point group. The setting manner of the second distance threshold is similar to that of the first distance threshold, and is not repeated.

For any target semantic information, after determining at least one ranging point group corresponding to the target semantic information, calculating a mean value of distances between ranging points in the ranging point group and the mobile robot for each ranging point group in the at least one ranging point group to determine the distance between the ranging point group and the mobile robot. In some embodiments, for each ranging point set, a distance between each ranging point in the ranging point set and the mobile robot (may be referred to as a third distance) may be calculated, and then a mean value of the third distances corresponding to each ranging point in the ranging point set is calculated to determine the distance between the ranging point set and the mobile robot. The calculation method of the distance between the ranging point and the mobile robot is described in detail above, and is not repeated.

For any target semantic information, after determining the distance between each ranging point group corresponding to the target semantic information and the mobile robot, the target semantic information corresponding to a specific ranging point group in at least one ranging point group corresponding to the target semantic information may be set as unknown, or the specific ranging point group in the at least one ranging point group may be deleted. In other words, in the solution of this example, only the target semantic information of the set of ranging points nearest to the mobile robot is retained. It will be appreciated that, due to the possible difference between the installation positions of the vision sensor and the ranging sensor on the mobile robot, when the ranging point in the first point cloud data is projected onto the pixel coordinate system corresponding to the vision data, there may be an error in the projection position of the ranging point in the pixel coordinate system (i.e., the coordinates of the projection point).

Fig. 2 shows a schematic view of a mobile robot according to an embodiment of the invention. As shown in fig. 2, the point a detected by the ranging sensor is actually a wall. However, since the installation positions of the vision sensor and the ranging sensor on the mobile robot are different, when the point a is projected to the pixel coordinate system corresponding to the vision data, the projection position of the ranging point in the pixel coordinate system coincides with the pixel point of which the semantic information is an obstacle, thereby causing the determined target semantic information to be wrong. In the scheme of the example, for each target semantic information, only target semantic information of a group of ranging point groups closest to the mobile robot in at least one ranging point group corresponding to the target semantic information is reserved. Therefore, the method is beneficial to avoiding the target semantic information error caused by the projection position error of the ranging point in the pixel coordinate system, and is beneficial to improving the accuracy of the second point cloud data. In summary, the solution helps to further improve the accuracy of the target map with semantic information.

Alternatively, the target map may be a grid map, and the map points of the target map may be grid points.

Optionally, storing the semantic information of each ranging point in the second point cloud data into a corresponding map point in the target map may include the following steps: for each map point, when the number of the ranging points corresponding to the map point in the current second point cloud data is a plurality of, storing semantic information with the largest repetition number in the target semantic information corresponding to each of the plurality of ranging points into the corresponding map point. In this embodiment, after each acquisition of visual data and corresponding first point cloud data, a set of semantic information is stored based only on the current visual data and the first point cloud data. Therefore, in this embodiment, the semantic information corresponding to each map point in the target map may be determined when the number of times of acquiring the visual data and the corresponding first point cloud data reaches the preset number.

Optionally, storing the semantic information of each ranging point in the second point cloud data into a corresponding map point in the target map may include the following steps: for each map point, when the number of the ranging points corresponding to the map point in the current second point cloud data is a plurality of, storing the target semantic information corresponding to each of the plurality of ranging points into the corresponding map point. In this embodiment, after visual data and corresponding first point cloud data are acquired each time, the number of semantic information stored in this time by each map point may be multiple. In this embodiment, the number of semantic information stored in each map point may be accumulated, and when the number of semantic information stored in the map point reaches a preset number, it is determined that the semantic information that occurs most repeatedly in the semantic information stored in the map point is the semantic information corresponding to the map point. For example, when the number of semantic information stored in a certain map point reaches 20 (10 times for a bed, 7 times for a wall, and 3 times for a curtain), the semantic information of the map point is determined to be a bed.

According to the technical scheme, the semantic information with the largest repeated occurrence among the semantic information stored in the map points is used as the semantic information corresponding to the map points, so that semantic information errors caused by single data errors can be avoided. The scheme is beneficial to improving the accuracy of semantic information corresponding to each map point in the target map.

Fig. 3 shows a schematic flow chart of a mapping method according to an embodiment of the invention. In this embodiment, the visual sensor is a depth camera, the visual data is a depth image, the ranging sensor is a radar sensor, the ranging points are radar points, the first point cloud data is a radar point cloud, and the target map is a 2d grid map. As shown in fig. 3, the method includes step S310, step S320, step S330, step S340, step S350, step S360, and step S370.

In step S310, the radar point cloud is projected into a pixel coordinate system corresponding to the depth image. In this step, the radar point cloud may first be turned into a camera coordinate system, and then turned into a pixel coordinate system according to the camera interior to obtain projection points corresponding one-to-one to the radar points in the radar point cloud.

In step S320, points that are not within the camera field of view are filtered out. In this step, the radar point cloud may be filtered according to the camera field of view to obtain the set of valid ranging points described above.

In step S330, target semantic information of the set of valid ranging points is determined. In this step, the pixel coordinates corresponding to each point (i.e., the coordinates of the projected point corresponding to each radar point in the set of valid ranging points in the pixel coordinate system) may be traversed, and a rectangle (i.e., the associated region) may be drawn centered on the point. For each radar point in the effective ranging point set, traversing each pixel in the rectangular frame corresponding to the radar point, and inquiring the semantics of the pixel. When the semantics of the pixel points in the frame are consistent (namely, the corresponding semantic information of each pixel point in the associated area is the same), the semantics of the point is set, otherwise, the point is deleted from the effective ranging point set, or the semantics of the point is set as unknown.

After the traversal is finished, step S340 is executed, and distance filtering is performed according to the semantic result. In the step, for the radar point in each effective ranging point set, judging whether the distance between the radar point and the mobile robot meets the distance requirement corresponding to the target semantic information corresponding to the ranging point, and deleting the radar point which does not meet the distance requirement from the effective ranging point set.

After the distance filtering is completed, step S350 is performed to remove the points with projection errors, so as to obtain the point cloud with semantics (i.e. the second point cloud data).

Next, step S360 is performed to update the second point cloud data into the 2d grid map.

In step S370, semantic information of each grid point in the 2d grid map may be determined according to the semantic information stored in the grid point, so as to obtain a semantic radar map (i.e., a target map with semantic information). Thus, the construction of the graph is completed.

The specific implementation manner of the above steps is described in detail above, and for brevity, will not be repeated here. It should be noted that, the present invention does not strictly limit the execution of the steps in the above embodiments, and one or more steps, such as step S320, step S340, and the like, may be omitted or the order of some steps may be adjusted, such as exchanging step S340 and step S350, for step S310, step S320, step S330, step S340, step S350, and step S360.

By the mapping method of any embodiment, the target map with semantic information can be accurately constructed. The target map obtained by the scheme can provide more accurate basis for moving and positioning the mobile robot, and is beneficial to improving the working efficiency and the working reliability of the mobile robot.

According to another aspect of the embodiment of the invention, a mapping device is also provided. The mapping device is applied to a mobile robot. The mobile robot includes a vision sensor and a ranging sensor. Fig. 4 shows a schematic block diagram of a mapping apparatus according to an embodiment of the invention. As illustrated in fig. 4, the mapping apparatus 400 includes an acquisition module 410, a fusion module 420, and a construction module 430.

The obtaining module 410 is configured to obtain semantic segmentation information and first point cloud data corresponding to the visual data, where the visual sensor collects the visual data for the target area, and the ranging sensor collects the first point cloud data for the target area.

The fusion module 420 is configured to fuse the first point cloud data and the semantic segmentation information to obtain second point cloud data with semantic information.

A construction module 430 is configured to construct a target map with semantic information based on the second point cloud data.

According to yet another aspect of an embodiment of the present invention, there is also provided a mobile robot. Fig. 5 shows a schematic block diagram of a mobile robot according to one embodiment of the invention. As shown in fig. 5, the mobile robot 500 includes: a processor 510 and a memory 520. The memory 520 stores a computer program, and the processor 510 is configured to execute the computer program to implement the mapping method described above. The mobile robot may be any one of a sweeping robot, a mopping robot, a sweeping and mopping robot, a transfer robot, and the like.

In some embodiments, the mobile robot to which the present invention relates refers to a mechanical device designed for cleaning, including but not limited to: a dust collector, a floor washing machine, a dust and water suction machine, a floor sweeping machine, a floor mopping machine, a sweeping and mopping integrated machine and the like.

For convenience of explanation, the mobile robot according to the embodiment of the present invention is exemplified as a cleaning robot.

Fig. 6 is a schematic block diagram of a cleaning robot in an embodiment. The cleaning robot includes a robot main body, a driving motor 610, a sensor unit 620, a controller 630, a cleaning member 640, a walking unit 650, a memory 660, a communication unit 670, an interaction unit 680, an energy storage unit 690, and the like.

The sensor unit 620 provided on the robot body may include at least one of the following sensors: radar sensors (e.g., lidar), collision sensors, distance sensors, fall sensors, counters, gyroscopes, vision sensors (e.g., depth cameras), etc. For example, the lidar is arranged on the top or the periphery of the robot body, and in operation, surrounding environmental information such as the distance and angle of the obstacle to the lidar can be obtained. In addition, a visual sensor such as a camera may be used instead of the laser radar, and the distance, angle, etc. of the obstacle with respect to the camera may be obtained by analyzing the obstacle in the image captured by the camera. The crash sensor includes, for example, a crash shell and a trigger sensing member; when the cleaning robot collides with the obstacle through the collision housing, the collision housing moves toward the inside of the cleaning robot, and compresses the elastic buffer member to play a role of buffering. After the collision housing moves a certain distance into the cleaning robot, the collision housing contacts the trigger sensing member, which is triggered to generate a signal that can be sent to the controller 630 in the robot body for processing. After collision with the obstacle, the cleaning robot is far away from the obstacle, and the collision shell moves back to the original position under the action of the elastic buffer piece. The distance sensor may specifically be an infrared detection sensor, and may be used to detect the distance of an obstacle to the distance sensor. The distance sensor may be provided at a side of the robot body so that a distance value of an obstacle located near the side of the cleaning robot to the distance sensor can be measured by the distance sensor. The distance sensor may be an ultrasonic distance sensor, a laser distance sensor, a depth sensor, or the like. The falling sensor is arranged at the bottom edge of the robot main body, and when the cleaning robot moves to the edge position of the ground, the falling sensor can detect the risk that the cleaning robot falls from a high place, so that corresponding anti-falling reaction is performed, for example, the cleaning robot stops moving, or moves in a direction away from the falling position, and the like. The inside of the robot main body is also provided with a counter and a gyroscope. The counter is used for detecting the distance length of the cleaning robot. The gyroscope is used for detecting the rotating angle of the cleaning robot, so that the direction of the cleaning robot can be determined.

The controller 630 is provided inside the robot main body, and the controller 630 is used to control the cleaning robot to perform a specific operation. The controller 630 may be, for example, a central processing unit (Central Processing Unit, CPU), a Microprocessor (Microprocessor), or the like. As shown in fig. 6, the controller 630 is electrically connected with the energy storage unit 690, the memory 660, the driving motor 610, the walking unit 650, the sensor unit 620, the interaction unit 680, the cleaning member 640, and the like to control these components.

The cleaning members 640 may be used to clean the floor, and the number of cleaning members 640 may be one or more. The cleaning member 640 includes, for example, a mop. The mop cloth comprises, for example, at least one of the following: the rotary mop, flat mop, roller mop, crawler mop, etc., are of course not limited thereto. The mop is arranged at the bottom of the robot main body, and can be specifically a position of the bottom of the robot main body, which is at a rear position. Taking the cleaning piece as a rotary mop for example, a driving motor 610 is arranged in the robot main body, two rotating shafts extend out of the bottom of the robot main body, and the mop is sleeved on the rotating shafts. The driving motor 610 may drive the rotation shaft to rotate, so that the rotation shaft drives the mop to rotate. The cleaning members 640 may also be side brushes, roll brushes, etc., without limitation.

The traveling unit 650 is a component related to movement of the cleaning robot, and the traveling unit 650 includes, for example, a driving wheel and a universal wheel. The universal wheels and the driving wheels are matched to realize the steering and the movement of the cleaning robot.

A memory 660 is provided on the robot body, and a program is stored on the memory 660, which when executed by the controller 630, implements a corresponding operation. The memory 660 is also used to store parameters for use by the cleaning robot. The Memory 660 includes, but is not limited to, magnetic disk Memory, compact disk read-Only Memory (CD-ROM), optical Memory, and the like.

A communication unit 670 is provided on the robot main body, the communication unit 670 for allowing the cleaning robot to communicate with external devices; for example with a terminal or with a base station. Wherein the base station is a cleaning device for use with a cleaning robot.

The interaction unit 680 is provided on the robot main body, and a user can interact with the cleaning robot through the interaction unit 680. The interaction unit 680 includes, for example, at least one of a touch screen, a switch button, a speaker, and the like. For example, the user can control the cleaning robot to start or stop by pressing a switch button.

An energy storage unit 690 is provided inside the robot body, the energy storage unit 690 being used to provide power for the cleaning robot.

The robot body is further provided with a charging part for acquiring power from an external device (e.g., a base station) to charge the energy storage unit 690 of the cleaning robot.

It should be understood that the cleaning robot described in fig. 6 is only one specific example of the embodiment of the present invention, and the cleaning robot of the embodiment of the present invention is not particularly limited. The cleaning robot of the embodiment of the invention can also be in other specific implementation modes. In other implementations, the cleaning robot may have more or fewer components than the cleaning robot shown in fig. 6; for example, the cleaning robot may include a clean water chamber for storing clean water and/or a dirt accommodating part for storing dirt, the cleaning robot may convey the clean water stored in the clean water chamber to the mop and/or the floor to wet the mop, and clean the floor based on the wet mop, and the cleaning robot may further collect dirt of the floor or sewage containing the dirt into the dirt accommodating part; the cleaning robot can also convey clean water stored in the clean water chamber to the cleaning piece so as to clean the cleaning piece, and dirty sewage containing dirt after cleaning the cleaning piece can also be conveyed into the dirty accommodating part.

According to yet another aspect of an embodiment of the present invention, there is also provided a computer-readable storage medium. The storage medium has stored therein a computer program/instruction which, when executed by a processor, implements the mapping method described above. The storage medium may include, for example, read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, or any combination of the preceding. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

The implementation structure, the working principle and the beneficial effects of the mapping device, the mobile robot and the computer readable storage medium are easily understood by those skilled in the art by reading the mapping method. For brevity, the description is omitted here.

Although the illustrative embodiments have been described herein with reference to the accompanying drawings, it is to be understood that the above illustrative embodiments are merely illustrative and are not intended to limit the scope of the present invention thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the invention. All such changes and modifications are intended to be included within the scope of the present invention as set forth in the appended claims.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

In the several embodiments provided by the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described device embodiments are merely illustrative, e.g., the division of elements is merely a logical function division, and there may be additional divisions of actual implementation, e.g., multiple elements or components may be combined or integrated into another device, or some features may be omitted, or not performed.

In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it should be appreciated that in order to streamline the invention and aid in understanding one or more of the various inventive aspects, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of the invention. However, the method of the present invention should not be construed as reflecting the following intent: i.e., the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.

It will be understood by those skilled in the art that all of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or units of any method or apparatus so disclosed, may be combined in any combination, except combinations where the features are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings), may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features but not others included in other embodiments, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.

Various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that some or all of the functions of some of the modules in the mapping apparatus and mobile robot according to the embodiments of the present invention may be implemented in practice using a microprocessor or Digital Signal Processor (DSP). The present invention can also be implemented as an apparatus program (e.g., a computer program and a computer program product) for performing a portion or all of the methods described herein. Such a program embodying the present invention may be stored on a computer readable medium, or may have the form of one or more signals. Such signals may be downloaded from an internet website, provided on a carrier signal, or provided in any other form.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, third, etc. do not denote any order. These words may be interpreted as names.

The foregoing description is merely illustrative of specific embodiments of the present invention and the scope of the present invention is not limited thereto, and any person skilled in the art can easily think about variations or substitutions within the scope of the present invention. The protection scope of the invention is subject to the protection scope of the claims.

Claims

1. The mapping method is applied to a mobile robot, and the mobile robot comprises a vision sensor and a ranging sensor; characterized by comprising the following steps:

Acquiring semantic segmentation information and first point cloud data corresponding to visual data, wherein the visual data are acquired by a visual sensor aiming at a target area, and the first point cloud data are acquired by a ranging sensor aiming at the target area;

fusing the first point cloud data and the semantic segmentation information to obtain second point cloud data with semantic information;

and constructing a target map with semantic information based on the second point cloud data.

2. The mapping method according to claim 1, wherein the semantic segmentation information includes semantic information corresponding to pixels in the visual data one by one;

the fusing the first point cloud data and the semantic segmentation information to obtain second point cloud data with semantic information includes:

projecting the ranging points in the first point cloud data to a pixel coordinate system corresponding to the visual data to obtain projection points corresponding to the ranging points in the first point cloud data one by one;

and determining target semantic information of at least part of ranging points in the first point cloud data at least based on semantic information corresponding to the pixel points at the coordinates of each projection point so as to obtain the second point cloud data.

3. The mapping method according to claim 2, wherein determining the target semantic information of at least part of the ranging points in the first point cloud data based at least on the semantic information corresponding to the pixel point at the coordinates of each projection point to obtain the second point cloud data includes:

determining target semantic information of an effective ranging point set at least based on semantic information corresponding to pixel points at coordinates of each projection point, wherein the effective ranging point set comprises ranging points meeting validity requirements in the first point cloud data;

Setting target semantic information corresponding to a ranging point with the target semantic information error in the effective ranging point set as unknown, or deleting the ranging point with the target semantic information error in the effective ranging point set to obtain the second point cloud data; correspondingly, the second point cloud data comprises ranging points of known target semantic information in the effective ranging point set, or the second point cloud data comprises the rest ranging points in the effective ranging point set.

4. A mapping method according to claim 3, wherein before determining the target semantic information of the set of valid ranging points based at least on the semantic information corresponding to the pixel point at the coordinates of each projection point, the method further comprises:

Deleting ranging points corresponding to projection points which are not in the visual field range of the visual sensor in the first point cloud data to obtain the effective ranging point set;

wherein the validity requirement includes: the ranging points correspond to projected points that are within the field of view of the vision sensor.

5. A mapping method according to claim 3, wherein the determining the target semantic information of the set of valid ranging points based at least on the semantic information corresponding to the pixel point at the coordinates of each projection point comprises:

for each ranging point in the set of valid ranging points,

Judging whether semantic information corresponding to each pixel point in an associated area is the same or not, wherein the associated area is an area with a preset size taking a projection point corresponding to the ranging point as a center;

when the semantic information corresponding to each pixel point in the associated area is the same, determining the semantic information corresponding to any pixel point in the associated area as the target semantic information of the ranging point; and/or the number of the groups of groups,

And when the semantic information corresponding to each pixel point in the associated area is not completely the same, determining that the target semantic information of the ranging point is wrong.

6. The mapping method according to any one of claims 3-5, wherein the setting, as unknown, the target semantic information corresponding to the ranging point with the target semantic information error in the valid ranging point set, or deleting the ranging point with the target semantic information error in the valid ranging point set, includes:

For each ranging point in the effective ranging point set, judging whether the distance between the ranging point and the mobile robot meets the distance requirement corresponding to the target semantic information corresponding to the ranging point;

And when the distance between the ranging point and the mobile robot does not meet the distance requirement, determining that the target semantic information of the ranging point is wrong, setting the target semantic information corresponding to the ranging point as unknown, or deleting the ranging point.

7. The mapping method according to any one of claims 3-5, wherein the setting, as unknown, the target semantic information corresponding to the ranging point with the target semantic information error in the valid ranging point set, or deleting the ranging point with the target semantic information error in the valid ranging point set, includes:

Traversing the ranging points in the effective ranging point set, and dividing the ranging points adjacent to the index and corresponding to the target semantic information into at least one ranging point group for any target semantic information;

For each ranging point group in the at least one ranging point group, calculating a mean value of distances between each ranging point in the ranging point group and the mobile robot to determine the distance between the ranging point group and the mobile robot;

Setting target semantic information corresponding to a specific ranging point group in the at least one ranging point group as unknown, or deleting the specific ranging point group in the at least one ranging point group, wherein the specific ranging point group is other ranging point groups except for the ranging point group with the minimum distance with the mobile robot;

The distance between any two adjacent ranging points in each ranging point group is smaller than or equal to a preset distance threshold, the distance between the ranging points corresponding to any two ranging point groups is larger than the preset distance threshold, and the ranging points with the target semantic information errors comprise the ranging points in the specific ranging point group.

8. The mapping method according to any one of claims 1-5, wherein the constructing a target map with semantic information based on the second point cloud data includes:

storing semantic information of each ranging point in the second point cloud data into a corresponding map point in the target map, wherein the semantic information of each ranging point is a group of semantic information;

for each map point within the target map,

When the number of the semantic information stored in the map points reaches a preset number, determining the semantic information with the largest repeated occurrence among the semantic information stored in the map points as the semantic information corresponding to the map points.

9. A mapping device is applied to a mobile robot, wherein the mobile robot comprises a vision sensor and a ranging sensor; characterized in that the device comprises:

The acquisition module is used for acquiring semantic segmentation information and first point cloud data corresponding to visual data, wherein the visual data are acquired by the visual sensor aiming at a target area, and the first point cloud data are acquired by the ranging sensor aiming at the target area;

The fusion module is used for fusing the first point cloud data and the semantic segmentation information to obtain second point cloud data with semantic information;

and the construction module is used for constructing a target map with semantic information based on the second point cloud data.

10. A mobile robot, comprising: a processor and a memory, the memory having stored therein a computer program for executing the computer program to implement the mapping method as claimed in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that a computer program/instruction is stored, which, when being executed by a processor, implements a mapping method as claimed in any one of claims 1-8.