CN115376109B - Obstacle detection method, obstacle detection device, and storage medium - Google Patents
Obstacle detection method, obstacle detection device, and storage medium Download PDFInfo
- Publication number
- CN115376109B CN115376109B CN202211311141.6A CN202211311141A CN115376109B CN 115376109 B CN115376109 B CN 115376109B CN 202211311141 A CN202211311141 A CN 202211311141A CN 115376109 B CN115376109 B CN 115376109B
- Authority
- CN
- China
- Prior art keywords
- image
- obstacle
- point cloud
- grid
- probability
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/762—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using clustering, e.g. of similar faces in social networks
Abstract
The application discloses barrier detection method, barrier detection device and storage medium, this barrier detection method includes: acquiring a point cloud image, a grid map and a detection image of an area to be detected; acquiring a panoramic segmentation result based on the detection image; fusing the panoramic segmentation semantics of the detection image with the point cloud image to obtain a point cloud semantic image; acquiring a first obstacle probability of each grid on a grid map; and fusing the point cloud semantic image with the grid map, and acquiring the probability of the fused obstacle of each grid by using the first obstacle probability of the grid and the number of projection points of the point cloud semantic image in the grid. By means of the method, the data of the multiple sensors can be fused, the algorithm complexity can be reduced, the probability of the obstacles is updated by considering the visual field range of the sensors, and the calculation efficiency and accuracy of obstacle detection are improved.
Description
Technical Field
The present application relates to the field of computer vision applications, and in particular, to an obstacle detection method, an obstacle detection apparatus, and a storage medium.
Background
With the improvement of automation degree, computer image processing can be applied to various occasions, and specifically can comprise robot vision, digital image communication, facial recognition, satellite image processing, resource exploration, agricultural planning, urban planning and the like.
In an application scenario, when the service robot moves on the ground, various objects existing in the actual environment can block the robot on the travel route of the robot, and the robot judges the obstacle information in the environment according to the map information transmitted by the sensor. However, the traditional map has low precision and low updating frequency, so that the robot cannot acquire environmental information in real time and comprehensively.
Compared with the defects of the traditional map, the method for mapping the point cloud information to the map can obtain more precise map information. Generally, a laser radar is used for synchronous positioning and mapping of a robot, but the laser radar only has point cloud information of one plane, and the problem of error mapping of noise points can occur in a map by directly mapping the point cloud to the map according to the characteristic that the point cloud changes along with the distance, so that the precision of the map is greatly influenced, the robot is difficult to sense environment information in an all-round manner, and whether obstacles exist in an actual scene or not can not be accurately judged when the robot selects a path.
Disclosure of Invention
The technical problem that this application mainly solved is how to improve efficiency and the accuracy that the barrier detected, to this, this application provides a barrier detection method, barrier detection device and storage medium.
In order to solve the technical problem, the application adopts a technical scheme that: there is provided an obstacle detection method including: acquiring a point cloud image, a grid map and a detection image of an area to be detected; acquiring a panoramic segmentation result based on the detection image; fusing the panoramic segmentation semantics of the detection image with the point cloud image to obtain a point cloud semantic image; acquiring a first obstacle probability of each grid on a grid map; and fusing the point cloud semantic image with the grid map, and acquiring the probability of the fused obstacle of each grid by using the first obstacle probability of the grid and the number of projection points of the point cloud semantic image in the grid.
The method comprises the following steps of fusing a panoramic segmentation semantic of a detection image with a point cloud image to obtain a point cloud semantic image, and comprises the following steps: acquiring camera internal parameters of a first camera for acquiring a detection image and camera internal parameters of a second camera for acquiring a point cloud image; acquiring projection matrixes of a first camera and a second camera based on camera internal parameters of the first camera and camera internal parameters of the second camera; and fusing the panoramic segmentation semantics of the detection image with the point cloud image based on the projection matrix to obtain a point cloud semantic image.
Wherein, the method also comprises: setting a preset neighborhood radius and a preset point threshold; and clustering all data points in the point cloud image, traversing all the data points, and marking the data points with the number of the related clustering points within the preset neighborhood radius being more than or equal to the preset point threshold as core points.
The method includes the steps of fusing a panoramic segmentation semantic of a detection image with a point cloud image to obtain a point cloud semantic image, and includes the following steps: and fusing the panoramic segmentation semantics of the detection image with core points in the point cloud image to obtain a point cloud semantic image.
Wherein, the method also comprises: adding all clustering points of the core point in a preset neighborhood radius into a set; traversing clustering points in the set, marking the clustering points with the number of the related clustering points in the preset neighborhood radius larger than or equal to a preset point threshold as core points, and marking the data points with the number of the related clustering points in the preset neighborhood radius smaller than the point threshold as boundary points; and marking data points except the core point and the boundary point in the point cloud image as noise points.
The method includes the steps of fusing a panoramic segmentation semantic of a detection image with a point cloud image to obtain a point cloud semantic image, and includes the following steps: and fusing the panoramic segmentation semantics of the detection image with core points and boundary points in the point cloud image to obtain a point cloud semantic image.
The method for acquiring the fusion obstacle probability of each grid by using the obstacle probability of the grid and the projection point quantity of the point cloud semantic image in the grid comprises the following steps: projecting the point cloud semantic image to a grid map, and acquiring the number of projection points of each grid on the grid map; acquiring a second obstacle probability corresponding to the point cloud semantic map based on the number of projection points of each grid; and calculating the fusion obstacle probability of each grid by using the first obstacle probability and the second obstacle probability of each grid.
Calculating the fusion obstacle probability of each grid by using the first obstacle probability and the second obstacle probability of each grid, wherein the calculation comprises the following steps: calculating a normalization coefficient based on the first obstacle probability and the second obstacle probability of each grid; and fusing the first obstacle probability and the second obstacle probability by using the normalization coefficient to obtain a fused obstacle probability.
Wherein, the method also comprises: acquiring first positioning information and a first fusion obstacle probability corresponding to a first frame point cloud image of a region to be detected, and second positioning information and a second fusion obstacle probability corresponding to a second frame point cloud image; acquiring a superposition view field area of the first frame point cloud image and the second frame point cloud image based on the first positioning information and the second positioning information; in the overlapping visual field region, a final fusion obstacle probability of each grid is calculated based on the first fusion obstacle probability and the second fusion obstacle probability of each grid.
Wherein, in the overlapping view field region, calculating a final fused obstacle probability for each grid based on the first fused obstacle probability and the second fused obstacle probability for each grid, including: in the overlapped visual field area, comparing the first fusion obstacle probability and the second fusion obstacle probability of the same grid; and calculating the first fusion obstacle probability and the second fusion obstacle probability with different probability values of the same grid by using a preset filtering equation to obtain the final fusion obstacle probability of the grid.
In order to solve the above technical problem, another technical solution adopted by the present application is: the obstacle detection device comprises an acquisition module, a segmentation module, a fusion module, a probability module and a detection module; the acquisition module is used for acquiring a point cloud image, a grid map and a detection image of an area to be detected; the segmentation module is used for acquiring a panoramic segmentation result based on the detection image, and the panoramic segmentation result comprises the panoramic segmentation semantics of each pixel point in the detection image; the fusion module is used for fusing the panoramic segmentation semantics of the detection image with the point cloud image to obtain a point cloud semantic image; the probability module is used for acquiring the probability of a first obstacle of each grid on the grid map; and the detection module is used for fusing the point cloud semantic image with the grid map, and acquiring the probability of the fused obstacle of each grid by using the first obstacle probability of the grid and the number of projection points of the point cloud semantic image in the grid.
In order to solve the above technical problem, another technical solution adopted by the present application is: an obstacle detection device is provided, which includes a processor and a memory, the memory is coupled to the processor, the memory stores program data, and the processor is used for executing the program data to realize the obstacle detection method.
In order to solve the above technical problem, another technical solution adopted by the present application is: there is provided a computer-readable storage medium storing program data for implementing the above-described obstacle detection method when the program data is executed.
The beneficial effect of this application is: different from the situation of the prior art, the obstacle detection method provided by the invention is applied to an obstacle detection device, and the obstacle detection device acquires a point cloud image, a grid map and a detection image of a to-be-detected area; acquiring a panoramic segmentation result based on the detection image; fusing the panoramic segmentation semantics of the detection image with the point cloud image to obtain a point cloud semantic image; acquiring a first obstacle probability of each grid on a grid map; and fusing the point cloud semantic image with the grid map, and acquiring the probability of the fused obstacle of each grid by using the first obstacle probability of the grid and the number of projection points of the point cloud semantic image in the grid. Through the mode, compared with conventional obstacle detection, the method for fusing the map obstacle probability based on the point cloud semantics and the grid map information and updating the probability based on the coincidence view field of the sensor can effectively fuse data among multiple sensors, reduce algorithm complexity, improve algorithm stability by utilizing multi-frame information, solve the problems that a map is not accurate enough and object information in an actual environment cannot be comprehensively obtained, and detect obstacles more accurately.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts. Wherein:
fig. 1 is a schematic flow chart of a first embodiment of a method of obstacle detection provided herein;
FIG. 2 is a schematic flow chart diagram illustrating an embodiment of a method step 13 for obstacle detection provided herein;
FIG. 3 is a schematic flow chart diagram illustrating one embodiment of a method step 15 for obstacle detection provided herein;
FIG. 4 is a schematic flow chart diagram illustrating one embodiment of method step 153 of obstacle detection provided herein;
FIG. 5 is a schematic flow chart diagram of a second embodiment of a method of obstacle detection provided herein;
FIG. 6 is a schematic flow chart diagram of a third embodiment of a method of obstacle detection provided herein;
FIG. 7 is a schematic illustration of obstacle update based on coincident fields of view in a method of obstacle detection provided herein;
FIG. 8 is a schematic flow chart diagram illustrating one embodiment of a method step 63 for obstacle detection provided herein;
fig. 9 is a schematic structural diagram of a first embodiment of an obstacle detection apparatus provided in the present application;
fig. 10 is a schematic structural diagram of a second embodiment of an obstacle detecting device provided in the present application;
FIG. 11 is a schematic diagram of an embodiment of a mobile robot provided herein;
FIG. 12 is a schematic structural diagram of an embodiment of a computer-readable storage medium provided in the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Referring to fig. 1, fig. 1 is a schematic flow chart of a first embodiment of a method for obstacle detection provided by the present application, the method including:
step 11: and acquiring a point cloud image, a grid map and a detection image of the area to be detected.
Optionally, the area to be detected is an initial image obtained by a sensor, and in this embodiment, taking path planning of the robot as an example, various objects on a traveling route of the robot prevent the robot from reaching a destination, so that the robot is required to detect the object and avoid an obstacle. In this embodiment, the area to be detected may be an area where a travel route of the robot from an initial location to a destination is located.
Specifically, the point cloud image includes point cloud data, which is data information of points on the surface of the object appearance obtained by the depth camera, and the point cloud map is a data matrix formed by the point cloud information. In this embodiment, the movement of the robot is mainly taken as an example, and the point cloud image is a matrix formed by data information of points on the surfaces of all objects on the travel route of the robot.
In particular, a grid map is a map that divides the environment into a series of grids, where each grid is given a possible value that represents the probability that the grid is occupied. Grid maps are typically constructed using lidar sensors. The detection image is obtained by using an RGB camera, and the image comprises image color data of an object and a background in a region to be detected and values of red, green and blue components of each pixel point of the image.
Step 12: and acquiring a panoramic segmentation result based on the detection image.
And the panoramic division result comprises the panoramic division semantics of each pixel point in the detection image. The panorama segmentation semantics are categories and instance labels to which each pixel point belonging to an instance target in the image belongs.
Specifically, panorama segmentation refers to performing semantic category prediction on each pixel point including a background in an image, and giving an instance label to the pixel point belonging to an instance target. In this embodiment, taking a robot as an example, mainly detects objects on the ground, and outputs a panoramic segmentation result of common objects in a family including the ground, including the category and the example label of the object.
Step 13: and fusing the panoramic segmentation semantics of the detection image and the point cloud image to obtain a point cloud semantic image.
In the embodiment of the application, the point cloud semantic image is obtained by assigning semantic information obtained by panoramic segmentation, namely the category of each pixel point to each point cloud in the point cloud image. Specifically, the obstacle detection device calibrates a camera for collecting a detection image and a camera for collecting a point cloud image so as to obtain a projection relationship between the two cameras, associates data points in the point cloud image with pixel points in the detection image according to the projection relationship, and finally fuses panoramic segmentation semantics of all the pixel points of the detection image into the point cloud image according to the association relationship of the two images so as to obtain a point cloud semantic image.
Optionally, in an embodiment, as shown in fig. 2, fig. 2 is a schematic flow chart of an embodiment of step 13, where step 13 may specifically include:
step 131: the camera internal reference of a first camera for collecting the detection image and the camera internal reference of a second camera for collecting the point cloud image are obtained.
Specifically, the internal parameters of the camera are related to the internal parameters of the camera, that is, the focal length, pixel size, and other data of the camera, and do not change with the position of the object. Camera intrinsic parameters involve a transformation from the imaging plane coordinate system to the pixel coordinate system, i.e. a transformation from the physical coordinates of the camera-captured image into the pixel coordinates of the image. The internal reference of the camera is usually acquired using a calibration method. The calibration mainly refers to shooting a calibration object with a known size by using a camera, and acquiring internal and external parameters of a camera model by utilizing a certain algorithm through establishing correspondence between a point with a known coordinate on the calibration object and an image pixel point of the calibration object. I.e. to implement the process of converting from the world coordinate system to the image coordinate system. In this embodiment, the original world coordinate system is replaced with the coordinate system of one of the cameras, and the conversion of the points in the first camera coordinate system and the second camera coordinate system can be realized.
Alternatively, in an embodiment, the camera parameters of the first and second cameras may be estimated using the Zhang calibration method.
Specifically, the calibration plate may be placed in a common field of view of two cameras, capturing multiple calibration plate images. The internal parameters of the two cameras are respectively calculated according to the position change of the calibration plate in the plurality of images by shooting the physical coordinates of the calibration plate in the images and the corresponding pixel coordinates in the images.
Step 132: the projection matrices of the first camera and the second camera are acquired based on camera intrinsic parameters of the first camera and camera intrinsic parameters of the second camera.
In particular, the projection matrix between the two cameras includes an appearance matrix between the first camera and the second camera. The external reference matrix comprises a rotation matrix R describing the direction of the coordinate axes of the first camera relative to the coordinate axes of the second camera and a translation matrix T describing the position of the origin of the second camera in the first camera coordinate system. The rotation matrix R and the translation matrix T together describe how the points are transformed in the two-camera coordinate system.
Step 133: and fusing the panoramic segmentation semantics of the detection image with the point cloud image based on the projection matrix to obtain a point cloud semantic image.
Specifically, the semantics of the pixel points in the detected image and the points in the point cloud image are fused by using the internal reference matrix of the two cameras and the external reference matrix R/T between the two cameras, so as to obtain the point cloud semantic image. The points in the point cloud semantic image are provided with point category and instance labels.
Step 14: a first obstacle probability for each grid on the grid map is obtained.
The first obstacle probability of each grid is the probability of judging whether the grid state is occupied or not. For example, the area where the robot travel path is located is divided into grids, a grid map is constructed through the sensors, and each grid probability represents the probability that the sensor considers that the grid has the obstacle.
Step 15: and fusing the point cloud semantic image with the grid map, and acquiring the probability of the fused obstacle of each grid by using the first obstacle probability of the grid and the number of projection points of the point cloud semantic image in the grid.
In the embodiment of the application, the first obstacle probability of the grid is the probability that the grid in a grid map constructed by the laser radar sensor of the robot through the acquired ground information is occupied, and specifically, the obstacle detection device converts points into a coordinate system of the ground through coordinate conversion according to the position information of each point in the point cloud semantic image, and projects the point into the grid map through the position point conversion. Thus, the association relation between the point cloud semantic image and the grid map is established, and each point in the point cloud semantic image is projected into the plane of the grid map according to the association relation. Based on the projection points within each grid and the probability that the grid in the grid map is occupied, they are fused using DS evidence theory to obtain a comprehensive probability that each grid has obstacles.
Optionally, in an embodiment, as shown in fig. 3, fig. 3 is a schematic flow chart of an embodiment of step 15, where step 15 may specifically include:
step 151: and projecting the point cloud semantic image to a grid map, and acquiring the number of projection points of each grid on the grid map.
Specifically, the number of projection points is the number of points in the point cloud semantic image projected to the grid map according to the association relationship between the point cloud semantic image and the grid map.
Step 152: and acquiring the second obstacle probability corresponding to the point cloud semantic map based on the number of the projection points of each grid.
Specifically, the second obstacle probability is related to the number of points included in each grid after the points in the point cloud semantic image are projected to the plane where the grid map is located according to the association relationship between the point cloud image and the grid map.
Specifically, the second obstacle probability may be defined by the following equation:
wherein, x represents the point number of the point cloud semantic image projected in each grid in the grid map, and N represents the obstacle judgment parameter related to the grid size in the grid map.
Step 153: and calculating the fusion obstacle probability of each grid by using the first obstacle probability and the second obstacle probability of each grid.
Specifically, in the present embodiment, the merged obstacle probability is a comprehensive obstacle probability obtained by the robot from the obstacle detection device, which merges the data of whether the grid has an obstacle or not with the multi-sensor determination unit.
Optionally, in an embodiment, as shown in fig. 4, fig. 4 is a schematic flowchart of an embodiment of step 153, where step 153 may specifically include:
step 1531: calculating a normalization coefficient based on the first and second obstacle probabilities for each grid.
The normalization coefficients are used for normalization of the data, and the data is mapped between [0,1] by linear transformation of the original data.
Specifically, the normalization coefficient can be defined by the following formula:
where pm and pd represent the first obstacle probability and the second obstacle probability, respectively.
Step 1532: and fusing the first obstacle probability and the second obstacle probability by using the normalization coefficient to obtain a fused obstacle probability.
Specifically, the fusion obstacle probability is calculated using the following formula:
specifically, the above two formulas belong to DS evidence theory, and multiple subjects can be fused by using interval estimation to deal with uncertainty problem. Taking obstacle detection of a robot as an example, the grid probability and the depth data of the detection area may be fused together to determine a comprehensive probability.
Optionally, in an embodiment, as shown in fig. 5, fig. 5 is a schematic flowchart of a second embodiment of the method for detecting an obstacle provided by the present application:
in the embodiment of the application, a circle in a certain range with a certain point as a central point is called a neighborhood of the certain point, the radius of the circle where the neighborhood is located is a neighborhood radius, and the preset point number threshold is the minimum sample number in the neighborhood range of the point. Specifically, a point cloud set in a point cloud image acquired by an obstacle detection device is input, and a neighborhood radius and a given point are set as the minimum number of core points within the neighborhood radius. And traversing all points in the point cloud image set to judge the core points, and outputting the core points and the boundary points to obtain a clustering result of the point cloud image.
Step 51: and setting a preset neighborhood radius and a preset point threshold.
Specifically, the preset point number threshold value is the minimum point number that makes a point to be determined a core point within a circle range that takes a preset neighborhood radius as a circle radius and takes the point as a center.
Step 52: and clustering all data points in the point cloud image, traversing all the data points, and marking the data points with the number of related clustering points in a preset neighborhood radius larger than or equal to the preset point threshold as core points.
Specifically, clustering is a process of grouping objects such as data points, and grouping similar objects into a group to form multiple classes. The clusters generated by clustering are a collection of a set of data objects that are similar to objects in the same cluster and distinct from objects in other clusters.
In the embodiment of the present application, traversing is performed on all points in an input point cloud image set, and first, it is determined whether the points have been classified (classified as core points, boundary points, or noise), if so, it is determined as next points, and if not, it is determined as core points.
Specifically, the core point is determined to calculate the number of midpoints in the range of a circle of which the radius is the preset neighborhood radius of the point and to mark the number of neighborhood points of the point as the number of neighborhood points of the point, if the number of the neighborhood points is larger than a preset point threshold value, the point is marked as the core point and is classified into a core point set C, and otherwise, the point is marked as a noise point.
Step 53: and adding all the clustering points of the core point in the preset neighborhood radius into the set.
Specifically, all points in the neighborhood of the point that has been determined to be the core point are classified into the boundary point set N.
Step 54: traversing the cluster points in the set, marking the cluster points with the number of the related cluster points in the preset neighborhood radius larger than or equal to the preset point threshold value as core points, and marking the data points with the number of the related cluster points in the preset neighborhood radius smaller than the preset point threshold value as boundary points.
Specifically, traversing all points in a boundary point set N, firstly judging whether the point is judged, if not, marking the points in the set N as p1, p2 and p3 \8230, marking the judged points as judged points, calculating the number of midpoints in a circle with the preset neighborhood radius as the radius of the point as the neighborhood point number of the point, if the neighborhood point number is more than or equal to a preset point threshold value, marking the point as a core point, classifying the point into a core point set C, classifying all points in the neighborhood range of the point into the boundary point set N, and otherwise, marking the point as a noise point. And judging all the points in the set N until each point in the point cloud image set is judged.
Step 55: and marking data points except the core point and the boundary point in the point cloud image as noise points.
Specifically, the core point set C and the boundary point N are output as a result of clustering the point cloud sets in the point cloud image. The noise point is a point which does not belong to any core point neighborhood range and the number of points in the core point neighborhood range is less than a preset point threshold value.
Optionally, the panoramic segmentation semantics of all the pixel points of the detected image are fused with the core points and the boundary points output after point cloud clustering in the point cloud image, so as to obtain the point cloud semantic image.
Referring to fig. 6, fig. 6 is a schematic flow chart of a third embodiment of the method for detecting an obstacle provided by the present application, the method including:
step 61: acquiring first positioning information and first fusion obstacle probability corresponding to a first frame point cloud image of a region to be detected, and second positioning information and second fusion obstacle probability corresponding to a second frame point cloud image.
Optionally, the positioning information corresponding to the point cloud image may be obtained by an instant positioning and mapping technique. The technology can be used for enabling the robot to complete positioning, mapping, path planning and the like in an unknown environment.
Specifically, the first frame point cloud image and the second frame point cloud image are point cloud images of two different time points acquired by the obstacle detection device, and due to the fact that the time points are different, positioning information, namely first positioning information and second positioning information, of corresponding cameras at different times and different fusion obstacle probabilities, namely first fusion obstacle probabilities and second fusion obstacle probabilities, of fusing multi-sensor data are obtained, the visual field of the cameras can have a certain offset angle, and the visual field of the cameras can have an overlapping area based on the offset angle.
Step 62: and acquiring a superposed view field area of the first frame point cloud image and the second frame point cloud image based on the first positioning information and the second positioning information.
Specifically, the overlapping view field is shown in fig. 7, and the overlapping view field is the overlapping position where the two cameras acquire images of the same object at different angles at different positions.
And step 63: in the overlapping visual field region, a final fusion obstacle probability of each grid is calculated based on the first fusion obstacle probability and the second fusion obstacle probability of each grid.
In the embodiment of the application, each grid in the overlapping view area range of the camera has the fusion obstacle probabilities from two different time points, and the fusion probabilities of the grids may be different, so that the two different fusion obstacle probabilities need to be fused through a filtering equation to obtain the final fusion obstacle probability of each grid.
Alternatively, the first and second fused obstacle probabilities for each grid in the region of overlapping fields of view may be fused using kalman filter equations to calculate a final fused obstacle probability for each grid.
Specifically, kalman filtering is based on the state space representation of a linear system, and the optimal estimation of the system state is obtained from output and input observation data. The optimal estimation can also be seen as a filtering process, since the observed data includes the effects of noise and interference in the system. The system state is a set of minimum parameters that summarize the effect of all past inputs and disturbances of the system on the system.
Optionally, in an embodiment, as shown in fig. 8, fig. 8 is a schematic flowchart of an embodiment of step 63, where step 63 may specifically include:
step 631: in the overlapping visual field region, the first and second fusion obstacle probabilities of the same grid are compared.
In the embodiment of the present application, taking the robot as an example, during the operation of the robot on the ground, at different time points, due to different positions of the robot, there may be an angular shift of the camera view, and therefore, a coincident view area may occur. In this coincident field of view region, two different probabilities of fusion obstacles may exist for the same grid.
Step 632: and calculating the first fusion obstacle probability and the second fusion obstacle probability with different probability values of the same grid by using a preset filtering equation to obtain the final fusion obstacle probability of the grid.
Different from the prior art, the obstacle detection method provided by the embodiment is applied to an obstacle detection device, and the obstacle detection device acquires a point cloud image, a grid map and a detection image of a to-be-detected area; acquiring a panoramic segmentation result based on the detection image; fusing the panoramic segmentation semantics of the detection image with the point cloud image to obtain a point cloud semantic image; acquiring a first obstacle probability of each grid on a grid map; and fusing the point cloud semantic image with the grid map, and acquiring the probability of the fused obstacle of each grid by using the first obstacle probability of the grid and the number of projection points of the point cloud semantic image in the grid. Through the mode, compared with conventional obstacle detection, the method for fusing the map obstacle probability based on the point cloud semantics and the grid map information and updating the probability based on the coincidence view field of the sensor can effectively fuse data among multiple sensors, reduce algorithm complexity, improve algorithm stability by utilizing multi-frame information, solve the problems that a map is not accurate enough and object information in an actual environment cannot be comprehensively obtained, and detect obstacles more accurately.
The method of the above embodiment may be implemented by using a detection device, and referring to fig. 9, fig. 9 is a schematic structural diagram of a first embodiment of the obstacle detection device provided in this application, and the obstacle detection device 90 may include an obtaining module 91, a dividing module 92, a fusing module 93, a probability module 94, and a detection module 95.
The acquisition module 91 is used for acquiring a point cloud image, a grid map and a detection image of an area to be detected; the segmentation module 92 is configured to obtain a panorama segmentation result based on the detection image, where the panorama segmentation result includes a panorama segmentation semantic of each pixel point in the detection image; the fusion module 93 is configured to fuse the panoramic segmentation semantic of the detection image with the point cloud image to obtain a point cloud semantic image; the probability module 94 is configured to obtain a first obstacle probability for each grid on the grid map; the detection module 95 is configured to fuse the point cloud semantic image with the grid map, and obtain a probability of a fused obstacle of each grid by using the first obstacle probability of the grid and the number of projection points of the point cloud semantic image in the grid.
Specifically, for a to-be-detected area transmitted by a sensor, a point cloud image, a grid map and a detection image of the to-be-detected area are acquired at first. And processing the image color data of the object and the background in the detected image by using a panoramic segmentation method to obtain the panoramic segmentation semantic category of each pixel point in the detected image. And clustering the point cloud in the point cloud image to facilitate subsequent fusion with the obtained panorama segmentation semantic category to obtain a point cloud semantic image. And aiming at each grid of the grid map, acquiring a first obstacle probability represented by each grid, fusing the first obstacle probability with a second obstacle probability related to the number of projection points for projecting the point cloud semantic image into each grid by utilizing the corresponding relation of the points, and acquiring a fused obstacle probability of each grid.
Referring to fig. 10, fig. 10 is a schematic structural diagram of a second embodiment of the obstacle detection device 100 provided in the present application, where the obstacle detection device 100 includes a memory 101 and a processor 102, the memory 101 is used for storing program data, and the processor 102 is used for executing the program data to implement the following method:
acquiring a point cloud image, a grid map and a detection image of an area to be detected; acquiring a panoramic segmentation result based on the detection image; fusing the panoramic segmentation semantics of the detection image with the point cloud image to obtain a point cloud semantic image; acquiring a first obstacle probability of each grid on a grid map; and fusing the point cloud semantic image with the grid map, and acquiring the probability of the fused obstacle of each grid by using the first obstacle probability of the grid and the number of projection points of the point cloud semantic image in the grid.
Referring to fig. 11, fig. 11 is a schematic structural diagram of a mobile robot according to an embodiment of the present disclosure. The mobile robot 110 includes an obstacle detection device 111, wherein the obstacle detection device may be the obstacle detection device shown in fig. 9 or fig. 10. The mobile robot may be a robot in multiple fields, such as a sweeping robot, a navigation robot, and a meal delivery robot.
In contrast, the mobile robot further includes various cameras and sensor devices for acquiring a point cloud image, a grid map, and a detection image of the area to be detected, which is not limited herein.
Referring to fig. 12, fig. 12 is a schematic structural diagram of an embodiment of a computer-readable storage medium 120 provided in the present application, where the computer-readable storage medium 120 stores program data 121, and when the program data 121 is executed by a processor, the method is implemented as follows:
acquiring a point cloud image, a grid map and a detection image of an area to be detected; acquiring a panoramic segmentation result based on the detection image; fusing the panoramic segmentation semantics of the detection image with the point cloud image to obtain a point cloud semantic image; acquiring a first obstacle probability of each grid on a grid map; and fusing the point cloud semantic image with the grid map, and acquiring the probability of the fused obstacle of each grid by using the first obstacle probability of the grid and the number of projection points of the point cloud semantic image in the grid.
Embodiments of the present application may be implemented in software functional units and may be stored in a computer readable storage medium when sold or used as a stand-alone product. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, a network device, or the like) or a processor (processor) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above description is only for the purpose of illustrating embodiments of the present application and is not intended to limit the scope of the present application, and all modifications of equivalent structures and equivalent processes, which are made by the contents of the specification and the drawings of the present application or are directly or indirectly applied to other related technical fields, are also included in the scope of the present application.
Claims (11)
1. An obstacle detection method, characterized by comprising:
acquiring a point cloud image, a grid map and a detection image of an area to be detected;
acquiring a panoramic division result based on the detection image, wherein the panoramic division result comprises the panoramic division semantics of each pixel point in the detection image;
fusing the panoramic segmentation semantics of the detection image with the point cloud image to obtain a point cloud semantic image;
acquiring a first obstacle probability of each grid on the grid map;
projecting the point cloud semantic image to the grid map to obtain the number of projection points of each grid on the grid map;
acquiring a second obstacle probability corresponding to the point cloud semantic map based on the number of projection points of each grid;
calculating a fused obstacle probability for each grid using the first obstacle probability and the second obstacle probability for each grid.
2. The obstacle detection method according to claim 1,
the method for fusing the panoramic segmentation semantics of the detection image and the point cloud image to obtain a point cloud semantic image comprises the following steps:
acquiring camera internal parameters of a first camera for acquiring the detection image and camera internal parameters of a second camera for acquiring the point cloud image;
acquiring projection matrixes of the first camera and the second camera based on camera intrinsic parameters of the first camera and camera intrinsic parameters of the second camera;
and fusing the panoramic segmentation semantics of the detection image with the point cloud image based on the projection matrix to obtain a point cloud semantic image.
3. The obstacle detection method according to claim 1 or 2,
the obstacle detection method includes:
setting a preset neighborhood radius and a preset point threshold;
clustering all data points in the point cloud image, traversing all the data points, and marking the data points with the related clustering point number more than or equal to the preset point threshold value in the preset neighborhood radius as core points;
the method for fusing the panoramic segmentation semantics of the detection image and the point cloud image to obtain the point cloud semantic image comprises the following steps:
and fusing the panoramic segmentation semantics of the detection image with the core points in the point cloud image to obtain the point cloud semantic image.
4. The obstacle detection method according to claim 3,
the obstacle detection method further includes:
adding all the clustering points of the core point in the preset neighborhood radius into a set;
traversing the cluster points in the set, marking the cluster points with the number of the related cluster points in the preset neighborhood radius larger than or equal to the preset point threshold value as the core points, and marking the data points with the number of the related cluster points in the preset neighborhood radius smaller than the preset point threshold value as boundary points;
marking data points in the point cloud image except the core point and the boundary point as noise points;
the method for fusing the panoramic segmentation semantics of the detection image and the point cloud image to obtain the point cloud semantic image comprises the following steps:
and fusing the panoramic segmentation semantics of the detection image with the core points and the boundary points in the point cloud image to obtain the point cloud semantic image.
5. The obstacle detection method according to claim 1,
the calculating a fused obstacle probability for each grid using the first obstacle probability and the second obstacle probability for each grid includes:
calculating a normalized coefficient based on the first and second obstacle probabilities for each grid;
and fusing the first obstacle probability and the second obstacle probability by using the normalization coefficient to obtain the fused obstacle probability.
6. The obstacle detection method according to claim 1,
the obstacle detection method further includes:
acquiring first positioning information and a first fusion obstacle probability corresponding to a first frame point cloud image of a region to be detected, and second positioning information and a second fusion obstacle probability corresponding to a second frame point cloud image;
acquiring a superposition view field area of the first frame point cloud image and the second frame point cloud image based on the first positioning information and the second positioning information;
calculating a final fused obstacle probability for each grid based on the first and second fused obstacle probabilities for each grid within the overlapping field of view region.
7. The obstacle detection method according to claim 6,
calculating a final fused obstacle probability for each grid based on the first fused obstacle probability and the second fused obstacle probability for each grid within the overlapping field of view region, including:
in the overlapped visual field area, comparing the first fusion obstacle probability and the second fusion obstacle probability of the same grid;
and calculating the first fusion obstacle probability and the second fusion obstacle probability with different probability values of the same grid by using a preset filtering equation to obtain the final fusion obstacle probability of the grid.
8. The obstacle detection device is characterized by comprising an acquisition module, a segmentation module, a fusion module, a probability module and a detection module; wherein the content of the first and second substances,
the acquisition module is used for acquiring a point cloud image, a grid map and a detection image of an area to be detected;
the segmentation module is used for acquiring a panoramic segmentation result based on the detection image, wherein the panoramic segmentation result comprises the panoramic segmentation semantics of each pixel point in the detection image;
the fusion module is used for fusing the panoramic segmentation semantics of the detection image with the point cloud image to obtain a point cloud semantic image;
the probability module is used for acquiring the probability of a first obstacle of each grid on the grid map;
the detection module is used for projecting the point cloud semantic image to the grid map, acquiring the number of projection points of each grid on the grid map, acquiring a second obstacle probability corresponding to the point cloud semantic map based on the number of projection points of each grid, and calculating the fusion obstacle probability of each grid by using the first obstacle probability and the second obstacle probability of each grid.
9. An obstacle detection apparatus, comprising a memory and a processor coupled to the memory;
wherein the memory is for storing program data and the processor is for executing the program data to implement the obstacle detection method of any one of claims 1 to 7.
10. A mobile robot comprising the obstacle detection device according to claim 8 or 9.
11. A computer storage medium for storing program data which, when executed by a computer, is used to implement the obstacle detection method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211311141.6A CN115376109B (en) | 2022-10-25 | 2022-10-25 | Obstacle detection method, obstacle detection device, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211311141.6A CN115376109B (en) | 2022-10-25 | 2022-10-25 | Obstacle detection method, obstacle detection device, and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115376109A CN115376109A (en) | 2022-11-22 |
| CN115376109B true CN115376109B (en) | 2023-03-24 |
Family
ID=84073173
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211311141.6A Active CN115376109B (en) | 2022-10-25 | 2022-10-25 | Obstacle detection method, obstacle detection device, and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115376109B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115981344B (en) * | 2023-02-06 | 2023-06-23 | 联通智网科技股份有限公司 | Automatic driving method and device |
| CN115797817B (en) * | 2023-02-07 | 2023-05-30 | 科大讯飞股份有限公司 | Obstacle recognition method, obstacle display method, related equipment and system |
| CN117541537A (en) * | 2023-10-16 | 2024-02-09 | 江苏星湖科技有限公司 | Space-time difference detection method and system based on all-scenic-spot cloud fusion technology |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113110451A (en) * | 2021-04-14 | 2021-07-13 | 浙江工业大学 | Mobile robot obstacle avoidance method with depth camera and single line laser radar fused |
Family Cites Families (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11566903B2 (en) * | 2018-03-02 | 2023-01-31 | Nvidia Corporation | Visualization of high definition map data |
| CN110032181B (en) * | 2019-02-26 | 2022-05-17 | 文远知行有限公司 | Method and device for positioning barrier in semantic map, computer equipment and storage medium |
| CN111190981B (en) * | 2019-12-25 | 2020-11-06 | 中国科学院上海微系统与信息技术研究所 | Method and device for constructing three-dimensional semantic map, electronic equipment and storage medium |
| CN111461245B (en) * | 2020-04-09 | 2022-11-04 | 武汉大学 | Wheeled robot semantic mapping method and system fusing point cloud and image |
| KR20210135389A (en) * | 2020-05-04 | 2021-11-15 | 현대자동차주식회사 | Apparatus for recognizing an obstacle, a vehicle system having the same and method thereof |
| US11967161B2 (en) * | 2020-06-26 | 2024-04-23 | Amazon Technologies, Inc. | Systems and methods of obstacle detection for automated delivery apparatus |
| CN112543938B (en) * | 2020-09-29 | 2022-02-08 | 华为技术有限公司 | Generation method and device of grid occupation map |
| CN112183476B (en) * | 2020-10-28 | 2022-12-23 | 深圳市商汤科技有限公司 | Obstacle detection method and device, electronic equipment and storage medium |
| CN112258618B (en) * | 2020-11-04 | 2021-05-14 | 中国科学院空天信息创新研究院 | Semantic mapping and positioning method based on fusion of prior laser point cloud and depth map |
| CN112859859B (en) * | 2021-01-13 | 2022-04-22 | 中南大学 | Dynamic grid map updating method based on three-dimensional obstacle object pixel object mapping |
| CN112785704A (en) * | 2021-01-13 | 2021-05-11 | 北京小马慧行科技有限公司 | Semantic map construction method, computer readable storage medium and processor |
| CN115147328A (en) * | 2021-03-29 | 2022-10-04 | 华为技术有限公司 | Three-dimensional target detection method and device |
| CN113670292B (en) * | 2021-08-10 | 2023-10-20 | 追觅创新科技(苏州)有限公司 | Map drawing method and device, sweeper, storage medium and electronic device |
| CN114391777B (en) * | 2022-01-07 | 2023-08-04 | 美智纵横科技有限责任公司 | Obstacle avoidance method and device for cleaning robot, electronic equipment and medium |
| CN114627073A (en) * | 2022-03-14 | 2022-06-14 | 一汽解放汽车有限公司 | Terrain recognition method, terrain recognition device, computer equipment and storage medium |
| CN114648654A (en) * | 2022-03-22 | 2022-06-21 | 北京航空航天大学合肥创新研究院(北京航空航天大学合肥研究生院) | Clustering method for fusing point cloud semantic categories and distances |
| CN114384920B (en) * | 2022-03-23 | 2022-06-10 | 安徽大学 | Dynamic obstacle avoidance method based on real-time construction of local grid map |
| CN114842106A (en) * | 2022-03-24 | 2022-08-02 | 北京石头创新科技有限公司 | Method and apparatus for constructing grid map, self-walking apparatus, and storage medium |
-
2022
- 2022-10-25 CN CN202211311141.6A patent/CN115376109B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113110451A (en) * | 2021-04-14 | 2021-07-13 | 浙江工业大学 | Mobile robot obstacle avoidance method with depth camera and single line laser radar fused |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115376109A (en) | 2022-11-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111462135B (en) | Semantic mapping method based on visual SLAM and two-dimensional semantic segmentation | |
| CN115376109B (en) | Obstacle detection method, obstacle detection device, and storage medium | |
| CN108932736B (en) | Two-dimensional laser radar point cloud data processing method and dynamic robot pose calibration method | |
| CN110988912B (en) | Road target and distance detection method, system and device for automatic driving vehicle | |
| CN114708585B (en) | Attention mechanism-based millimeter wave radar and vision fusion three-dimensional target detection method | |
| CN112667837A (en) | Automatic image data labeling method and device | |
| CN111882612A (en) | Vehicle multi-scale positioning method based on three-dimensional laser detection lane line | |
| CN108648194B (en) | Three-dimensional target identification segmentation and pose measurement method and device based on CAD model | |
| CN105160649A (en) | Multi-target tracking method and system based on kernel function unsupervised clustering | |
| CN113447923A (en) | Target detection method, device, system, electronic equipment and storage medium | |
| CN112562093B (en) | Object detection method, electronic medium, and computer storage medium | |
| CN115049700A (en) | Target detection method and device | |
| CN113205604A (en) | Feasible region detection method based on camera and laser radar | |
| CN111383204A (en) | Video image fusion method, fusion device, panoramic monitoring system and storage medium | |
| CN115273034A (en) | Traffic target detection and tracking method based on vehicle-mounted multi-sensor fusion | |
| CN114578328B (en) | Automatic calibration method for spatial positions of multiple laser radars and multiple camera sensors | |
| CN115311512A (en) | Data labeling method, device, equipment and storage medium | |
| CN114926808A (en) | Target detection and tracking method based on sensor fusion | |
| KR102490521B1 (en) | Automatic calibration through vector matching of the LiDAR coordinate system and the camera coordinate system | |
| CN112150448B (en) | Image processing method, device and equipment and storage medium | |
| CN112017259B (en) | Indoor positioning and image building method based on depth camera and thermal imager | |
| CN112488022A (en) | Panoramic monitoring method, device and system | |
| CN114611635B (en) | Object identification method and device, storage medium and electronic device | |
| CN111899277A (en) | Moving object detection method and device, storage medium and electronic device | |
| CN114782496A (en) | Object tracking method and device, storage medium and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |