WO2022088611A1 - 障碍检测方法、装置、电子设备、存储介质及计算机程序 - Google Patents

障碍检测方法、装置、电子设备、存储介质及计算机程序 Download PDF

Info

Publication number
WO2022088611A1
WO2022088611A1 PCT/CN2021/085927 CN2021085927W WO2022088611A1 WO 2022088611 A1 WO2022088611 A1 WO 2022088611A1 CN 2021085927 W CN2021085927 W CN 2021085927W WO 2022088611 A1 WO2022088611 A1 WO 2022088611A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
detected
position information
specific object
area
Prior art date
Application number
PCT/CN2021/085927
Other languages
English (en)
French (fr)
Inventor
张展鹏
成慧
Original Assignee
深圳市商汤科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳市商汤科技有限公司 filed Critical 深圳市商汤科技有限公司
Priority to KR1020217034499A priority Critical patent/KR20220060500A/ko
Priority to JP2021562323A priority patent/JP2023503747A/ja
Publication of WO2022088611A1 publication Critical patent/WO2022088611A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/80—Geometric correction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/07—Target detection

Definitions

  • the embodiments of the present disclosure relate to the technical field of path planning, and in particular, to an obstacle detection method, an apparatus, an electronic device, a storage medium, and a computer program.
  • Path planning in the field of robot navigation refers to: first inform the robot of the starting point and target point of walking, and then plan a reasonable path for the robot to walk according to the known map information in an indoor environment with obstacles.
  • a reasonable path should at least meet the following conditions: the path is a relatively short path from the starting point to the target point; the path can avoid obstacles in the known map to a great extent.
  • the embodiments of the present disclosure provide at least an obstacle detection method, an apparatus, an electronic device, a storage medium, and a computer program.
  • a first aspect of the embodiments of the present disclosure provides an obstacle detection method, the obstacle detection method includes: acquiring an image to be detected; subjecting the to-be-detected image to object detection to obtain position information of the object in the to-be-detected image;
  • the to-be-detected image is semantically segmented, and the location information of the ground area in the to-be-detected image is acquired; the obstacle area is acquired based on the location information of the object and the location information of the ground area.
  • the obstacle area is jointly determined based on the position information of the object and the position information of the ground area, and the position information of the ground area in the image to be detected is obtained based on semantic segmentation, it is possible to improve the accuracy of the determined ground area. Therefore, the obstacle area with high accuracy can be obtained, and the obstacle analysis of the environment can be carried out effectively.
  • the acquiring the image to be detected includes: acquiring calibration parameters of the camera, wherein the calibration parameters include distortion parameters; acquiring a captured target image, and correcting the target image based on the distortion parameters to obtain the to-be-detected image.
  • the calibration parameter further includes a transformation parameter;
  • the performing semantic segmentation of the image to be detected to obtain the location information of the ground area in the image to be detected includes: inputting the image to be detected into the first a deep neural network to obtain the semantic labels of all pixels in the image to be detected, wherein the semantic labels include ground labels and background labels; based on the transformation parameters, the ground labels in the image to be detected correspond to The pixel points of are transformed into the ground area in the camera coordinate system, and the position information of the ground in the camera coordinate system is obtained.
  • the ground area and the background area in the image to be detected are distinguished by the semantic label of the first deep neural network, and the information of the image to be detected is further projected to the camera coordinate system through the transformation parameters, which is beneficial to reflect the difference between the ground area, the background area and the The distance relationship of the camera.
  • the calibration parameter further includes a transformation parameter
  • the object in the image to be detected includes a non-specific object
  • the object detection is performed on the image to be detected
  • the position information of the object in the image to be detected is obtained , further comprising: inputting the image to be detected into a second deep neural network to obtain a contour frame of a non-specific object in the image to be detected; based on the transformation parameters, transforming the contour frame of the non-specific object into The outline frame of the camera coordinate system is obtained, and the position information of the non-specific object in the camera coordinate system is obtained.
  • the non-specific objects in the image to be detected are semantically segmented through the second deep neural network to demarcate the contour boxes of the non-specific objects, and the shape of the non-specific objects is represented by the contour boxes, which is conducive to sticking to the non-specific objects in path planning. Get a plan path with high practicability.
  • the calibration parameter further includes a transformation parameter
  • the object in the image to be detected includes a specific object
  • the object detection is performed on the image to be detected
  • the position information of the object in the image to be detected is obtained
  • the method includes: inputting the image to be detected into a third deep neural network to obtain image position information of a specific object in the image to be detected; based on the transformation parameter, transforming the image position information of the object into The position information of the camera coordinate system.
  • object detection is performed on the specific object in the image to be detected through the third deep neural network, so as to obtain the position information of the specific object.
  • the inputting the to-be-detected image into a third deep neural network and acquiring the image position information of a specific object in the to-be-detected image includes: inputting the to-be-detected image into the third deep neural network The network is used to obtain the bounding box of the specific object in the to-be-detected image; and the image position information of the specific object is calculated based on the diagonal coordinates of the bounding box.
  • the obstacle detection method further includes: forming a current planning map based on the position information of the object and the position information of the ground, wherein the current planning map includes a drivable area and includes the object The obstacle area is obtained; the planned route is obtained based on the drivable area of the current planning map.
  • the driving area is obtained through the position information on the ground, and the obstacle area in the driving area is obtained through the position information of the object, thereby generating a current planning map for path planning.
  • the obstacle detection method further includes: in response to the fact that the object in the to-be-detected image includes a non-specific object, determining that the obstacle area includes a contour frame of the non-specific object in the camera coordinate system and/or, in response to the situation that the object in the image to be detected includes a specific object, obtain the category information of the specific object; generate the specific object according to the position information and category information of the specific object.
  • a method for forming an obstacle area is proposed, which is beneficial to quickly generate an obstacle area in the driving area according to the object frame; and for a specific object, considering the category information of the specific object can enable the obstacle detection device to determine whether it needs to be close to the specific object.
  • planning which is conducive to improving the practicability of the planning path.
  • a second aspect of an embodiment of the present disclosure provides an obstacle detection device, the obstacle detection device includes: a camera part configured to acquire an image to be detected; an object detection part configured to perform object detection on the image to be detected, Obtain the position information of the object in the to-be-detected image; the semantic segmentation part is configured to perform semantic segmentation on the to-be-detected image to obtain the location information of the ground area in the to-be-detected image; the obstacle detection part is configured to be based on The position information of the object and the position information of the ground area are used to obtain an obstacle area and a planned path to avoid the obstacle area.
  • the camera part is configured to acquire calibration parameters of the camera, wherein the calibration parameters include distortion parameters; acquire a captured target image, and correct the target image based on the distortion parameters, Obtain the to-be-detected image.
  • the calibration parameters further include transformation parameters;
  • the semantic segmentation part is configured to input the image to be detected into a first deep neural network, and acquire semantic labels of all pixels in the image to be detected , wherein the semantic label includes a ground label and a background label; based on the transformation parameters, transform the pixel points corresponding to the ground label in the to-be-detected image into a ground area in the camera coordinate system to obtain the The location information of the ground area in the camera coordinate system.
  • the calibration parameter further includes a transformation parameter
  • the object in the image to be detected includes a non-specific object
  • the object detection part is further configured to input the image to be detected into a second deep neural network, Obtain the outline frame of the non-specific object in the to-be-detected image; based on the transformation parameters, transform the outline frame of the non-specific object into the outline frame of the camera coordinate system, and obtain the non-specific object in the The position information of the camera coordinate system.
  • the calibration parameter further includes a transformation parameter
  • the object in the image to be detected includes a specific object
  • the object detection part is further configured to input the image to be detected into a third deep neural network, and obtain Image position information of a specific object in the to-be-detected image; based on the transformation parameter, transform the image position information of the object into position information of the specific object in the camera coordinate system.
  • the object detection part is further configured to input the image to be detected into the third deep neural network, and obtain a bounding frame of a specific object in the image to be detected;
  • the diagonal coordinates calculate the image position information of the specific object.
  • the apparatus further includes: a path planning part configured to form a current planning map based on the position information of the object and the position information of the ground, wherein the current planning map includes a drivable area, and an obstacle area including the object; obtaining a planned route based on the drivable area of the current planned map.
  • a path planning part configured to form a current planning map based on the position information of the object and the position information of the ground, wherein the current planning map includes a drivable area, and an obstacle area including the object; obtaining a planned route based on the drivable area of the current planned map.
  • the apparatus further includes: an area determination part configured to determine that the obstacle area includes the non-specific object in the camera in response to the fact that the object in the to-be-detected image includes a non-specific object The area corresponding to the outline frame of the coordinate system; and/or, configured to obtain the category information of the specific object in response to the fact that the object in the to-be-detected image includes a specific object; according to the position information of the specific object and The category information generates an object frame of the specific object in the camera coordinate system; wherein, the obstacle area includes an area corresponding to the specific object in the object frame of the camera coordinate system.
  • a third aspect of the embodiments of the present disclosure provides an electronic device, including a mutually coupled memory and a processor, where the processor is configured to execute program instructions stored in the memory to implement the obstacle detection method in the first aspect.
  • a fourth aspect of the embodiments of the present disclosure provides a computer-readable storage medium on which program instructions are stored, and when the program instructions are executed by a processor, the obstacle detection method in the first aspect above is implemented.
  • a fifth aspect of the embodiments of the present disclosure provides a computer program, including computer-readable code, when the computer-readable code is executed in an electronic device, the processor in the electronic device executes the program to implement the above-mentioned first aspect The obstacle detection method in .
  • the obstacle detection device obtains the image to be detected; performs object detection on the image to be detected to obtain the position information of the object in the image to be detected; performs semantic segmentation on the image to be detected to obtain the position information of the ground area in the image to be detected; and the position information of the ground area to obtain the obstacle area.
  • the above solution can perform obstacle analysis on the environment.
  • FIG. 1 is a schematic flowchart of an embodiment of an obstacle detection method provided by an embodiment of the present disclosure
  • FIG. 2 is a schematic flowchart of an embodiment of step S11 in the obstacle detection method shown in FIG. 1;
  • FIG. 3 is a schematic diagram of a frame of an image to be detected including an object frame provided by an embodiment of the present disclosure
  • FIG. 4 is a schematic flowchart of an embodiment of step S12 in the obstacle detection method shown in FIG. 1;
  • FIG. 5 is a schematic flowchart of another embodiment of step S12 in the obstacle detection method shown in FIG. 1;
  • FIG. 6 is a schematic flowchart of an embodiment of step S13 in the obstacle detection method shown in FIG. 1;
  • FIG. 7 is a schematic frame diagram of an embodiment of an obstacle detection device provided by an embodiment of the present disclosure.
  • FIG. 8 is a schematic diagram of a framework of an embodiment of an electronic device provided by an embodiment of the present disclosure.
  • FIG. 9 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium provided by an embodiment of the present disclosure.
  • Sweeping robots are an important part of home smart terminals. While intelligent sweeping robots need to locate their own position, they also need to sense the information of surrounding objects for path planning and avoid obstacles, such as shoes, chair feet, and socks. etc., to achieve an efficient cleaning process. Therefore, obstacle detection analysis is an important technical component in the path planning of intelligent sweepers.
  • sweeping robots There are generally two types of sweeping robots: 1) sweeping robots based on single-line lidar; 2) sweeping robots based on binocular or monocular cameras.
  • single-line lidar solution the advantage is that it covers a large angular range, and the general single-line lidar can achieve a real-time 360-degree scanning range.
  • single-line lidar can only scan a single horizontal plane, so it cannot completely cover obstacles of different heights. For example, socks cannot be detected when they are close to the ground.
  • the use of multi-line lidar can alleviate this problem to a certain extent, but the current cost of multi-line lidar is high, and it is not suitable for use in consumer-grade products such as home sweepers.
  • this method has certain positional constraints between the two camera sensors of the binocular camera (for example, the two cameras must be separated by a certain distance and at the same height in the horizontal direction), and this scheme is relatively small for textures, such as white walls The scene adaptability is not good.
  • the embodiment of the present disclosure proposes a solution using a monocular camera, which integrates the target detection technology and An obstacle analysis method for semantic segmentation techniques.
  • an embodiment of the present disclosure proposes an obstacle detection method that can be applied to the realization of a sweeping robot in a home place.
  • the obstacle detection method proposed in the embodiments of the present disclosure can also be applied to other robots, such as cleaning robots, lawn mowing robots, outdoor delivery robots, etc., which will not be repeated here.
  • FIG. 1 is a schematic flowchart of an embodiment of an obstacle detection method provided by an embodiment of the present disclosure.
  • the execution subject of the obstacle detection method in the embodiment of the present disclosure may be an obstacle detection apparatus.
  • the obstacle detection method may be executed by a terminal device, a server, or other processing equipment, wherein the obstacle detection apparatus may be a user equipment (User Equipment, UE ), mobile devices, user terminals, terminals, cellular phones, wireless phones, Personal Digital Assistant (PDA), handheld devices, computing devices, in-vehicle devices, wearable devices, etc.
  • the obstacle detection method may be implemented by the processor invoking computer-readable instructions stored in the memory.
  • the obstacle detection method according to the embodiment of the present disclosure may include the following steps:
  • Step S11 Acquire an image to be detected.
  • the obstacle detection device acquires the to-be-detected image of the environment where the robot is located.
  • the image to be detected may be acquired by a camera part mounted on the robot, and the camera part may specifically be a monocular red-green-blue (RGB) camera.
  • the camera model is fixed at the fixed position of the robot, generally set at the front position of the robot running direction, and can obtain the environmental image in the forward direction of the robot, that is, the camera field of view needs to cover the ground and the possible obstacles that need to be analyzed.
  • Step S12 Perform object detection on the image to be detected, and obtain position information of the object in the image to be detected.
  • the obstacle detection apparatus performs object detection on the image to be detected, and obtains position information of the object in the image to be detected.
  • the object in the image to be detected is the obstacle; through the position information of the object, the obstacle detection device can generate an effective obstacle area.
  • the objects of the embodiments of the present disclosure can be classified into specific objects and non-specific objects.
  • specific objects are objects with relatively fixed shapes preset by the staff in advance, such as slippers, paper balls, cans, etc.
  • non-specific objects are objects with variable shapes, such as table legs, wires, etc.
  • Step S13 Semantically segment the image to be detected, and obtain location information of the ground area in the image to be detected.
  • the obstacle detection apparatus performs ground detection on the image to be detected through the semantic segmentation part.
  • the semantic segmentation part in this step and the above-mentioned semantic segmentation part for detecting non-specific objects may be the same part or different parts.
  • the input of the semantic segmentation part is the image to be detected after correction based on the calibrated distortion parameters, and the output is a number of pixels marked as the ground in the image to be detected, and the ground area composed of several pixels.
  • Step S14 Obtain the obstacle area based on the position information of the object and the position information of the ground.
  • the obstacle detection apparatus obtains the above-mentioned ground information and object information through the fusion part.
  • the input information of the fusion part includes: (1) the information obtained by the semantic segmentation part, that is, relative to the camera, the area belonging to the ground in the current area and the position and shape of non-specific objects; (2) the information obtained by the object detection part, that is, relative to the camera.
  • the camera runs the position and class of a specific object ahead.
  • the obstacle detection device can form a current planning map according to the above ground information and object information, wherein the current planning map can be a two-dimensional map or a three-dimensional map.
  • the obstacle area is jointly determined based on the position information of the object and the position information of the ground area, and the position information of the ground area in the image to be detected is obtained based on semantic segmentation, it is possible to improve the accuracy of the determined ground area. Therefore, the obstacle area with high accuracy can be obtained, and the obstacle analysis of the environment can be carried out effectively.
  • the obstacle detection device may also preprocess the image to be detected to improve the accuracy of the image to be detected. Please refer to FIG. 2 for the specific preprocessing process.
  • FIG. 2 shows the obstacle detection method shown in FIG. 1
  • Step S11 is a schematic flowchart of an embodiment. As shown in Figure 2, step S11 specifically includes the following steps:
  • Step S21 Obtain calibration parameters of the camera, wherein the calibration parameters include distortion parameters.
  • the obstacle detection device needs to calibrate the camera.
  • the obstacle detection device can calibrate the internal and external parameters of the camera part by Zhang Zhengyou's calibration method.
  • the internal parameters may include, but are not limited to, the focal length and distortion parameters of the camera
  • the external parameters may include, but are not limited to, the homography matrix of the camera.
  • the distortion parameters include tangential distortion and radial distortion. The radial distortion occurs in the process of converting the camera coordinate system to the image physical coordinate system, and the tangential distortion occurs in the camera manufacturing process, because the plane of the photosensitive element is not parallel to the plane of the lens.
  • the homography matrix is the transformation relationship of the projection mapping between the image physical coordinate system and the camera coordinate system.
  • the internal and external parameters of the camera part can also be calibrated by other calibration methods, such as Meng Hu's plane calibration method, Wu Yihong's parallel circle calibration method, and the like.
  • Step S22 acquiring the captured target image, and correcting the target image based on the distortion parameter to obtain an image to be detected.
  • the tangential distortion and the radial distortion may cause partial deformation of the image to be detected partially captured by the camera. Therefore, after the obstacle detection device acquires the target image, it needs to correct the target image based on the distortion parameters, so as to reduce the influence of the image distortion caused by the tangential distortion and radial distortion of the camera itself, which is beneficial to improve the accuracy of subsequent path planning.
  • the obstacle detection device can directly detect the object through the object detection part.
  • FIG. 4 is a schematic flowchart of an embodiment of step S12 in the obstacle detection method shown in FIG. 1 .
  • the calibration parameters also include transformation parameters.
  • step S12 specifically includes the following steps:
  • Step S31 Input the image to be detected into a third deep neural network, and obtain image position information of a specific object in the image to be detected.
  • the object detection part includes a third deep neural network and a geometric projection part.
  • the input of the third deep neural network is the image to be detected corrected based on the calibrated distortion parameters, and the output is the image to be detected containing the image position information of the specific object.
  • the image position information of a specific object is represented by the object frame in FIG. 3 .
  • the particular object shown in Figure 3 includes two balls of paper and a can.
  • step S31 can be implemented in the following ways: inputting the image to be detected into a third deep neural network to obtain the bounding box of the specific object in the image to be detected; calculating the image position information of the specific object based on the diagonal coordinates of the bounding box .
  • the third deep neural network pre-trains the specific object in the embodiments of the present disclosure, and the third deep neural network can identify the position and category of the specific object, which is embodied in the form of an object frame.
  • the object box may be composed of a bounding box surrounding a specific object and a semantic label, and the semantic label indicates the category of the specific object corresponding to the bounding box.
  • Step S32 Based on the transformation parameters, transform the image position information of the object into the position information of the specific object in the camera coordinate system.
  • the obstacle detection apparatus transforms the image position information of the object into the position information of the specific object in the camera coordinate system based on the transformation parameters, and reflects it in the form of a bounding box.
  • the transformation parameters are the external parameters of the calibrated camera part, including but not limited to the homography matrix of the camera.
  • the obstacle detection device uses the geometric projection part to take the midpoint position between the upper left corner coordinate and the lower right corner coordinate of the bounding box, or the focus position between the upper right corner coordinate and the lower left corner coordinate as the specific object in the image , and then combined with the pre-calibrated homography matrix to calculate the position of a specific object relative to the camera.
  • the ground area and the background area in the image to be detected are distinguished by the semantic label of the first deep neural network, and the information of the image to be detected is further projected to the camera coordinate system through the transformation parameters, which is beneficial to reflect the difference between the ground area, the background area and the The distance relationship of the camera.
  • the obstacle detection apparatus marks the category information of the specific object on the bounding box to form the object frame of the specific object as shown in FIG. 3 .
  • the category information of specific objects is conducive to planning a planning path that is closer to real life and has higher practicability. For example, when a specific object belongs to shoes and other objects that will not affect the robot, the planned path can be planned closely to the specific object; when the specific object belongs to more dangerous objects such as desk lamps and electric heaters, the planned path can be combined with Specific objects are planned at a certain distance to prevent hazards during robot operation.
  • the obstacle detection device can detect them by means of semantic segmentation. It is worth noting that the process of detecting non-specific objects by means of semantic segmentation can be detected by the above-mentioned object detection part, or by another method.
  • the semantic segmentation part is used for detection. Please refer to FIG. 5 for the specific detection process.
  • FIG. 5 is a schematic flowchart of another embodiment of step S12 in the obstacle detection method shown in FIG. 1 .
  • the calibration parameters also include transformation parameters, and the objects in the image to be detected include non-specific objects.
  • step S12 specifically includes the following steps:
  • Step S41 Input the image to be detected into the second deep neural network, and obtain the outline frame of the non-specific object in the image to be detected.
  • the input of the second deep neural network is an image to be detected corrected based on the calibrated distortion parameters
  • the output is an image to be detected that includes image position information of a non-specific object.
  • the image position information of the non-specific object is embodied by the outline frame surrounding the non-specific object.
  • Step S42 Based on the transformation parameters, transform the outline frame of the non-specific object into the outline frame in the camera coordinate system, and obtain the position information of the non-specific object in the camera coordinate system.
  • the obstacle detection apparatus transforms the image position information of the non-specific object into the position information of the non-specific object in the camera coordinate system based on the homography matrix, and reflects it in the form of an outline frame.
  • the above-mentioned third deep neural network can only output the object frame of a specific object, wherein the object frame is a rectangular frame surrounding the specific object; the recognition principle of the second deep neural network is to identify the edge points of non-specific objects, and then put a number of These edge points are combined and connected to form a closed edge line, that is, a contour box around a non-specific object.
  • the outline frame of a non-specific object can better reflect the specific shape information of the object, which is beneficial to the robot's running path to be planned closely to the non-specific object, so as to improve the practicability of the robot's planning path. .
  • the non-specific objects in the image to be detected are semantically segmented through the second deep neural network to demarcate the contour boxes of the non-specific objects, and the shape of the non-specific objects is represented by the contour boxes, which is conducive to sticking to the non-specific objects in path planning. Get a plan path with high practicability.
  • FIG. 6 is a schematic flowchart of an embodiment of step S13 in the obstacle detection method shown in FIG. 1 .
  • the calibration parameters also include transformation parameters.
  • step S13 specifically includes the following steps:
  • Step S51 Input the image to be detected into the first deep neural network, and acquire semantic labels of all pixels in the image to be detected, wherein the semantic labels include ground labels and background labels.
  • the semantic segmentation part includes a fully convolutional first deep neural network and a geometric projection part.
  • the input of the first deep neural network is the image to be detected corrected based on the calibrated distortion parameters, and the output is the semantic label of each pixel in the image to be detected.
  • semantic segmentation part can also be used to segment the outline frame of non-specific objects, that is, the above-mentioned second deep neural network and the first deep neural network in this step can be the same deep neural network, which will not be repeated here.
  • semantic tags specifically include ground tags and background tags. When a certain pixel is identified as a ground pixel, the semantic label is marked as 1; when a certain pixel is identified as a background pixel, the semantic label is marked as 0.
  • Step S52 Based on the transformation parameters, transform the pixel points corresponding to the ground label in the image to be detected into the ground area in the camera coordinate system, and obtain the position information of the ground in the camera coordinate system.
  • the obstacle detection apparatus projects each ground pixel point with a semantic label of 1 from the image space to the camera space based on the homography matrix, and obtains the position information of the ground pixel point in the camera coordinate system. Then, the obstacle detection device combines the projected ground pixels into the ground area of the camera coordinate system, and the remaining areas are background areas. Among them, the ground area is the driving area of the robot.
  • the obstacle detection device obtains the image to be detected; performs object detection on the image to be detected to obtain the position information of the object in the image to be detected; performs semantic segmentation on the image to be detected to obtain the position information of the ground in the image to be detected; The position information and the position information of the ground area obtain the obstacle area.
  • the obstacle detection device can automatically identify and mark the ground area and object position in the to-be-detected image by performing object detection and semantic segmentation on the to-be-detected image, wherein the ground area is the driving area of the robot, and the detected object When the position appears in the driving area, it can be analyzed that the object is an obstacle, so that the obstacle can be analyzed effectively in the environment, and the path planning can be further carried out according to the analysis result.
  • the writing order of each step does not mean a strict execution order but constitutes any limitation on the implementation process, and the specific execution order of each step should be based on its function and possible Internal logic is determined.
  • the ground area and the background area in the image to be detected are distinguished by the semantic label of the first deep neural network, and the information of the image to be detected is further projected to the camera coordinate system through the transformation parameters, which is beneficial to reflect the difference between the ground area, the background area and the The distance relationship of the camera.
  • the obstacle detection method may further include: forming a current planning map based on the position information of the object and the position information of the ground, wherein the current planning map includes a drivable area and an obstacle area including the object; The drivable area gets the planned route.
  • the driving area is obtained through the position information on the ground, and the obstacle area in the driving area is obtained through the position information of the object, thereby generating a current planning map for path planning.
  • the current planning map includes drivable areas and obstacle areas for objects.
  • the obstacle area is an area generated based on the position information of the object, and the drivable area is a part of the ground area excluding the obstacle area.
  • the area enclosed by the contour frame of the camera coordinate system of the non-specific object can be regarded as the location of the non-specific object, so this area is the obstacle area ( or a part of the obstacle area);
  • the category information of the specific object can be obtained through the above-mentioned third deep neural network (or other methods), and then according to the position information and category of the specific object
  • the information generates the object frame of the specific object in the camera coordinate system, and the area enclosed by the object frame of the specific object in the camera coordinate system can be regarded as the location of the specific object, so this area is the obstacle area (or a part of the obstacle area).
  • the size of the corresponding object frame may be different. This is considering that different specific objects have different effects on the robot, so they will correspond to different sizes of object frames so that the robot can plan more Precise path; for example, when a specific object belongs to shoes and other objects that will not affect the robot, the size of the object frame is small, that is, the obstacle area is small, and the planned path can be planned close to the specific object; In the case of more dangerous objects such as desk lamps and electric heaters, the size of the object frame is large, that is, the obstacle area is large, and the planned path can be planned at a certain distance from the specific object to prevent the robot from causing danger during operation. In the subsequent path planning process, considering the category information of specific objects is conducive to planning a planning path that is closer to real life and has higher practicability.
  • the obstacle detection device can obtain the drivable area of the robot in the driving area, and then input the drivable area into the trajectory planning part.
  • the obstacle detection device inputs the end point information through the trajectory planning part, and obtains the planned path to avoid the obstacle area.
  • the obstacle detection method may further include: in response to the fact that the object in the image to be detected includes a non-specific object, determining that the obstacle area includes an area corresponding to the contour frame of the non-specific object in the camera coordinate system.
  • the obstacle detection method may further include: in response to a situation in which the object in the image to be detected includes a specific object, acquiring category information of the specific object; generating the position information and category information of the specific object in the camera coordinate system The object frame; wherein, the obstacle area includes the area corresponding to the object frame of the specific object in the camera coordinate system.
  • a method for forming an obstacle area is proposed, which is beneficial to quickly generate an obstacle area in the driving area according to the object frame; and for a specific object, considering the category information of the specific object can enable the obstacle detection device to determine whether it needs to be close to the specific object.
  • planning which is conducive to improving the practicability of the planning path.
  • FIG. 7 is a schematic diagram of a framework of an embodiment of an obstacle detection apparatus provided by an embodiment of the present disclosure.
  • the obstacle detection device 70 includes:
  • the camera part 71 is configured to acquire the image to be detected.
  • the object detection part 72 is configured to perform object detection on the image to be detected, and obtain position information of the object in the image to be detected.
  • the semantic segmentation part 73 is configured to perform semantic segmentation on the image to be detected, and obtain location information of the ground area in the image to be detected.
  • the obstacle detection section 74 is configured to acquire the obstacle area based on the position information of the object and the position information of the ground area.
  • the camera part 71 is configured to acquire calibration parameters of the camera, wherein the calibration parameters include distortion parameters; acquire a captured target image, and correct the target image based on the distortion parameters to obtain an image to be detected.
  • the calibration parameters further include transformation parameters;
  • the semantic segmentation part 73 is configured to input the image to be detected into the first deep neural network, and obtain semantic labels of all pixels in the image to be detected, wherein the semantic labels include ground label and background label; based on the transformation parameters, transform the pixels corresponding to the ground label in the image to be detected into the ground area in the camera coordinate system, and obtain the position information of the ground area in the camera coordinate system.
  • the calibration parameter further includes a transformation parameter
  • the object in the image to be detected includes a non-specific object
  • the object detection part 72 is further configured to input the image to be detected into the second deep neural network, and obtain the non-specific object in the image to be detected.
  • the outline frame of the object; based on the transformation parameters, the outline frame of the non-specific object is transformed into the outline frame in the camera coordinate system, and the position information of the non-specific object in the camera coordinate system is obtained.
  • the calibration parameters further include transformation parameters
  • the object in the image to be detected includes a specific object
  • the object detection part 72 is further configured to input the image to be detected into a third deep neural network, and obtain the specific object in the image to be detected.
  • Image position information based on the transformation parameters, transform the image position information of the object into the position information of a specific object in the camera coordinate system.
  • the object detection part 72 is further configured to input the image to be detected into the third deep neural network to obtain the bounding box of the specific object in the image to be detected; calculate the image position of the specific object based on the diagonal coordinates of the bounding box information.
  • the apparatus further includes: a path planning part configured to form a current planning map based on the position information of the object and the position information of the ground, wherein the current planning map includes a drivable area and an obstacle area including the object; Get the planned route from the drivable area of the current planning map.
  • the apparatus further includes: an area determination part configured to, in response to a situation in which the object in the image to be detected includes a non-specific object, determine that the obstacle area includes an area corresponding to the contour frame of the non-specific object in the camera coordinate system; Or, it is configured to obtain category information of the specific object in response to the fact that the object in the image to be detected includes a specific object; generate an object frame of the specific object in the camera coordinate system according to the position information and category information of the specific object; wherein the obstacle area includes The area corresponding to a specific object in the object frame of the camera coordinate system.
  • the system for implementing the solutions of the embodiments of the present disclosure may include the following parts: a camera part, an object detection part, a ground segmentation part (corresponding to the semantic segmentation part in the above-mentioned embodiment), and a fusion part (corresponding to the above-mentioned semantic segmentation part)
  • the obstacle detection part in the embodiment may include the following parts: a camera part, an object detection part, a ground segmentation part (corresponding to the semantic segmentation part in the above-mentioned embodiment), and a fusion part (corresponding to the above-mentioned semantic segmentation part)
  • the obstacle detection part in the embodiment and the trajectory planning part.
  • the camera part can receive the current image and distribute the image to the object detection part and the semantic segmentation part respectively.
  • the object detection part detects all the objects in the set A in the picture, and outputs the object category and the position of the object relative to the current robot.
  • the ground segmentation part labels the pixels in the image for the input image, and outputs the area that belongs to the flat ground.
  • the fusion part outputs a map of the area where the sweeping robot can travel. According to this map, the trajectory planning part plans the trajectory of obstacle avoidance, and finally the system control part completes the execution of the obstacle avoidance trajectory.
  • the camera part needs to be fixed at a fixed position of the sweeping robot to determine whether the field of view of the camera part can cover the ground and possible obstacles that need to be analyzed. After fixing the camera part, there needs to be a calibration process.
  • the internal parameter I of the camera part (including focal length and distortion parameters) is calibrated
  • the external parameter E (here the external parameter refers to the homography matrix between the camera imaging plane and the two planes on the ground) is calibrated.
  • the calibration here can use Zhang Zhengyou's calibration method.
  • the input of the object detection part can be the corrected image based on the calibrated distortion parameters, and the output is the category and position of a specific type of obstacle.
  • a specific type of obstacle is an object with a relatively fixed shape (eg, slippers, paper balls, cans, etc.) that are formulated before development by the system developer.
  • the object detection part can include a deep neural network M1 and a geometric projection part G.
  • the input of neural network A is the picture, and the output is the bounding box of the detected object.
  • the geometric projection part uses the midpoint between the lower left corner and the lower right corner of the bounding box as the position of the object in the image, combined with pre-calibration
  • a good external parameter E can calculate the position of the object relative to the camera part.
  • the input of the ground segmentation part is the image rectified based on the calibrated distortion parameters, and the output is the semantic label of each pixel in the input image.
  • the semantic label is equal to 0 or 1.
  • it means that the pixel is the ground, otherwise it is the background.
  • the ground segmentation part can be implemented by a fully convolutional deep network M2. After obtaining the semantic label of each pixel, through the geometric part G, each pixel with semantic ground can be projected from the image space to the camera space (ie, the coordinates relative to the camera).
  • the fusion part can determine the following information: 1) The information obtained by the ground segmentation part, that is, which areas are currently on the ground relative to the camera part; 2) The information obtained by the object detection part, that is, relative to the camera part, where are the obstacles ahead? object. These two aspects of information are in the camera space system, so the two aspects of information are comprehensively analyzed, and the fusion part obtains which areas are drivable within the range covered by the current camera part.
  • the drivable area can be described using a local map. By inputting the local map into the trajectory planning part, the trajectory of obstacle avoidance can be obtained.
  • this solution is used to analyze obstacles, so as to know which obstacles are in front of the robot and which are walkable areas, and feed back this information to the planning control system, thereby Achieve obstacle avoidance.
  • FIG. 8 is a schematic diagram of a framework of an embodiment of an electronic device provided by an embodiment of the present disclosure.
  • the electronic device 80 includes a memory 81 and a processor 82 coupled to each other, and the processor 82 is configured to execute program instructions stored in the memory 81 to implement the steps in any of the above-mentioned embodiments of the obstacle detection method.
  • the electronic device 80 may include, but is not limited to, a microcomputer and a server.
  • the electronic device 80 may also include mobile devices such as a notebook computer and a tablet computer, which are not limited herein.
  • the processor 82 is configured to control itself and the memory 81 to implement the steps in any of the above-mentioned embodiments of the obstacle detection method.
  • the processor 82 may also be referred to as a central processing unit (Central Processing Unit, CPU).
  • the processor 82 may be an integrated circuit chip with signal processing capability.
  • the processor 82 may also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
  • a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
  • the processor 82 may be jointly implemented by an integrated circuit chip.
  • FIG. 9 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium provided by an embodiment of the present disclosure.
  • the computer-readable storage medium 90 stores program instructions 901 that can be executed by the processor, and the program instructions 901 are used to implement the steps in any of the above-mentioned embodiments of the obstacle detection method.
  • the functions or included parts of the apparatus may be used to execute the methods described in the above method embodiments, and the specific implementation may refer to the descriptions of the above method embodiments. No longer.
  • Embodiments of the present disclosure may further provide a computer program, including computer-readable code, when the computer-readable code is executed in an electronic device, the processor in the electronic device executes the program for implementing any of the above The steps in the embodiment of the obstacle detection method.
  • the disclosed method and apparatus may be implemented in other manners.
  • the device implementations described above are only illustrative.
  • the division of parts or units is only a logical function division. In actual implementation, there may be other divisions.
  • units or components may be combined or integrated. to another system, or some features can be ignored, or not implemented.
  • the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, which may be in electrical, mechanical or other forms.
  • each functional unit in each embodiment of the embodiments of the present disclosure may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.
  • the above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.
  • the integrated unit if implemented as a software functional unit and sold or used as a stand-alone product, may be stored in a computer-readable storage medium.
  • the technical solutions of the embodiments of the present disclosure may be embodied in the form of software products in essence, or the parts that contribute to the related technology, or all or part of the technical solutions, and the computer software products are stored in a storage medium. , including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the methods of the various implementations of the embodiments of the present disclosure.
  • the aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes .
  • the obstacle area is jointly determined based on the position information of the object and the position information of the ground area, and the position information of the ground area in the image to be detected is obtained based on semantic segmentation, it is possible to improve the determination accuracy.
  • the accuracy of the ground area, and then the obstacle area with high accuracy can be obtained, so that the obstacle analysis of the environment can be carried out effectively.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Biophysics (AREA)
  • General Engineering & Computer Science (AREA)
  • Biomedical Technology (AREA)
  • Mathematical Physics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computational Linguistics (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Image Analysis (AREA)
  • Geometry (AREA)

Abstract

本公开实施例公开了一种障碍检测方法、装置、电子设备、存储介质及计算机程序，该障碍检测方法包括：获取待检测图像；将待检测图像进行物体检测，获取待检测图像中物体的位置信息；将待检测图像进行语义分割，获取待检测图像中地面区域的位置信息；基于物体的位置信息以及地面区域的位置信息获取障碍区域。

Description

障碍检测方法、装置、电子设备、存储介质及计算机程序
相关申请的交叉引用
本专利申请要求2020年10月28日提交的中国专利申请号为202011172403.6，申请人为深圳市商汤科技有限公司，申请名称为“一种障碍检测方法、装置、电子设备以及存储介质”的优先权，该申请的全文以引用的方式并入本申请中。
技术领域
本公开实施例涉及路径规划技术领域，特别是涉及一种障碍检测方法、装置、电子设备、存储介质及计算机程序。
背景技术
近年来，随着移动机器人的快速发展，如何进行障碍物检测和避让是体现机器人智能化水平的重要标准，良好的避障功能是移动机器人安全行走的重要保障，而如何进行良好的避障就涉及到对机器人的路径规划。
机器人导航领域的路径规划是指：首先告知机器人行走的起始点和目标点，然后在存在障碍物的室内环境中根据已知地图信息规划出一条合理的路径供机器人行走。其中，合理的路径应当至少满足以下条件：路径是从起始点到达目标点的相对较短的路径；该路径能够极大程度的避开在已知地图中的障碍物。
发明内容
本公开实施例至少提供一种障碍检测方法、装置、电子设备、存储介质及计算机程序。
本公开实施例第一方面提供了一种障碍检测方法，所述障碍检测方法包括：获取待检测图像；将所述待检测图像进行物体检测，获取所述待检测图像中物体的位置信息；将所述待检测图像进行语义分割，获取所述待检测图像中地面区域的位置信息；基于所述物体的位置信息以及所述地面区域的位置信息获取障碍区域。
因此，由于基于所述物体的位置信息以及所述地面区域的位置信息，共同确定障碍区域，而待检测图像中地面区域的位置信息是基于语义分割获取的，使得能够提高确定的地面区域的准确性，进而能够获取到准确度 高的障碍区域，进而能够有效地对环境进行障碍物分析。
在一些实施例中，所述获取待检测图像，包括：获取相机的标定参数，其中，所述标定参数包括畸变参数；获取拍摄的目标图像，并基于所述畸变参数对所述目标图像进行矫正，得到所述待检测图像。
因此，通过畸变参数矫正目标图像，得到待检测图像，有利于提高路径规划的准确性。
在一些实施例中，所述标定参数还包括变换参数；所述将所述待检测图像进行语义分割，获取所述待检测图像中地面区域的位置信息，包括：将所述待检测图像输入第一深度神经网络，获取所述待检测图像中所有像素点的语义标签，其中，所述语义标签包括地面标签和背景标签；基于所述变换参数，将所述待检测图像中所述地面标签对应的像素点变换为在所述相机坐标系的地面区域，得到所述地面在所述相机坐标系的位置信息。
因此，通过第一深度神经网络的语义标签将待检测图像中的地面区域和背景区域区分开，进一步将待检测图像的信息通过变换参数投影到相机坐标系，有利于体现地面区域、背景区域与相机的距离关系。
在一些实施例中，所述标定参数还包括变换参数，所述待检测图像中物体包括非特定物体；所述将所述待检测图像进行物体检测，获取所述待检测图像中物体的位置信息，还包括：将所述待检测图像输入第二深度神经网络，获取所述待检测图像中非特定物体的轮廓框；基于所述变换参数，将所述非特定物体的轮廓框变换为在所述相机坐标系的轮廓框，得到所述非特定物体在所述相机坐标系的位置信息。
因此，通过第二深度神经网络对待检测图像中的非特定物体进行语义分割，以标定非特定物体的轮廓框，通过轮廓框表示非特定物体的形状，有利于在路径规划中紧贴非特定物体得到实用性高的规划路径。
在一些实施例中，所述标定参数还包括变换参数，所述待检测图像中物体包括特定物体；所述将所述待检测图像进行物体检测，获取所述待检测图像中物体的位置信息，包括：将所述待检测图像输入第三深度神经网络，获取所述待检测图像中特定物体的图像位置信息；基于所述变换参数，将所述物体的图像位置信息变换为所述特定物体在所述相机坐标系的位置信息。
因此，通过第三深度神经网络对待检测图像中的特定物体进行物体检测，得到特定物体的位置信息。
在一些实施例中，所述将所述待检测图像输入第三深度神经网络，获取所述待检测图像中特定物体的图像位置信息，包括：将所述待检测图像输入所述第三深度神经网络，获取所述待检测图像中特定物体的包围框；基于所述包围框的对角坐标计算所述特定物体的图像位置信息。
因此，提供通过包围框的对角坐标计算特定物体的位置信息的计算方法。
在一些实施例中，所述障碍检测方法还包括：基于所述物体的位置信息以及所述地面的位置信息形成当前规划地图，其中，所述当前规划地图包括可行驶区域，以及包括所述物体的障碍区域；基于所述当前规划地图的可行驶区域获取规划路径。
因此，通过地面的位置信息获得行驶区域，再通过物体的位置信息获得行驶区域中的障碍区域，从而生成当前规划地图，用于进行路径规划。
在一些实施例中，所述障碍检测方法还包括：响应于所述待检测图像中物体包括非特定物体的情况，确定所述障碍区域包括所述非特定物体在所述相机坐标系的轮廓框所对应的区域；和/或，响应于所述待检测图像中物体包括特定物体的情况，获取所述特定物体的类别信息；根据所述特定物体的位置信息和类别信息生成所述特定物体在相机坐标系的物体框；其中，所述障碍区域包括所述特定物体在所述相机坐标系的物体框所对应的区域。
因此，提出一种形成障碍区域的方法，有利于根据物体框快速生成行驶区域中的障碍区域；而对于特定物体，考虑特定物体的类别信息能够使障碍检测装置决定是否需要紧贴特定物体进行路径规划，有利于提高规划路径的实用性。
本公开实施例第二方面提供了一种障碍检测装置，所述障碍检测装置包括：相机部分，被配置为获取待检测图像；物体检测部分，被配置为将所述待检测图像进行物体检测，获取所述待检测图像中物体的位置信息；语义分割部分，被配置为将所述待检测图像进行语义分割，获取所述待检测图像中地面区域的位置信息；障碍检测部分，被配置为基于所述物体的位置信息以及所述地面区域的位置信息获取障碍区域以及避开所述障碍区域的规划路径。
在一些实施例中，所述相机部分，被配置为获取相机的标定参数，其中，所述标定参数包括畸变参数；获取拍摄的目标图像，并基于所述畸变参数对所述目标图像进行矫正，得到所述待检测图像。
在一些实施例中，所述标定参数还包括变换参数；所述语义分割部分，被配置为将所述待检测图像输入第一深度神经网络，获取所述待检测图像中所有像素点的语义标签，其中，所述语义标签包括地面标签和背景标签；基于所述变换参数，将所述待检测图像中所述地面标签对应的像素点变换为在所述相机坐标系的地面区域，得到所述地面区域在所述相机坐标系的位置信息。
在一些实施例中，所述标定参数还包括变换参数，所述待检测图像中物体包括非特定物体；所述物体检测部分，还被配置为将所述待检测图像输入第二深度神经网络，获取所述待检测图像中所述非特定物体的轮廓框；基于所述变换参数，将所述非特定物体的轮廓框变换为在所述相机坐标系的轮廓框，得到所述非特定物体在所述相机坐标系的位置信息。
在一些实施例中，所述标定参数还包括变换参数，所述待检测图像中物体包括特定物体；所述物体检测部分，还被配置为将所述待检测图像输入第三深度神经网络，获取所述待检测图像中特定物体的图像位置信息；基于所述变换参数，将所述物体的图像位置信息变换为所述特定物体在所述相机坐标系的位置信息。
在一些实施例中，所述物体检测部分，还被配置为将所述待检测图像输入所述第三深度神经网络，获取所述待检测图像中特定物体的包围框；基于所述包围框的对角坐标计算所述特定物体的图像位置信息。
在一些实施例中，所述装置还包括：路径规划部分，被配置为基于所述物体的位置信息以及所述地面的位置信息形成当前规划地图，其中，所述当前规划地图包括可行驶区域，以及包括所述物体的障碍区域；基于所述当前规划地图的可行驶区域获取规划路径。
在一些实施例中，所述装置还包括：区域确定部分，被配置为响应于所述待检测图像中物体包括非特定物体的情况，确定所述障碍区域包括所述非特定物体在所述相机坐标系的轮廓框所对应的区域；和/或，被配置为响应于所述待检测图像中物体包括特定物体的情况，获取所述特定物体的类别信息；根据所述特定物体的位置信息和类别信息生成所述特定物体在相机坐标系的物体框；其中，所述障碍区域包括所述特定物体在所述相机坐标系的物体框所对应的区域。
本公开实施例第三方面提供了一种电子设备，包括相互耦接的存储器和处理器，处理器用于执行存储器中存储的程序指令，以实现上述第一方面中的障碍检测方法。
本公开实施例第四方面提供了一种计算机可读存储介质，其上存储有程序指令，程序指令被处理器执行时实现上述第一方面中的障碍检测方法。
本公开实施例第五方面提供了一种计算机程序，包括计算机可读代码，当所述计算机可读代码在电子设备中运行时，所述电子设备中的处理器执行用于实现上述第一方面中的障碍检测方法。
上述方案，障碍检测装置获取待检测图像；将待检测图像进行物体检测，获取待检测图像中物体的位置信息；将待检测图像进行语义分割，获取待检测图像中地面区域的位置信息；基于物体的位置信息以及地面区域的位置信息获取障碍区域。上述方案，能够对环境进行障碍物分析。
应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，而非限制本公开实施例。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，这些附图示出了符合本公开的实施例，并与说明书一起用于说明本公开实施例的技术方 案。
图1是本公开实施例提供的障碍检测方法一实施例的流程示意图；
图2是图1所示障碍检测方法中步骤S11一实施例的流程示意图；
图3是本公开实施例提供的包括物体框的待检测图像的框架示意图；
图4是图1所示障碍检测方法中步骤S12一实施例的流程示意图；
图5是图1所示障碍检测方法中步骤S12另一实施例的流程示意图；
图6是图1所示障碍检测方法中步骤S13一实施例的流程示意图；
图7是本公开实施例提供的障碍检测装置一实施例的框架示意图；
图8是本公开实施例提供的电子设备一实施例的框架示意图；
图9是本公开实施例提供的计算机可读存储介质一实施例的框架示意图。
具体实施方式
下面结合说明书附图，对本公开实施例的方案进行详细说明。
以下描述中，为了说明而不是为了限定，提出了诸如特定系统结构、接口、技术之类的具体细节，以便透彻理解本公开实施例。
本文中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。此外，本文中的“多”表示两个或者多于两个。另外，本文中术语“至少一种”表示多种中的任意一种或多种中的至少两种的任意组合，例如，包括A、B、C中的至少一种，可以表示包括从A、B和C构成的集合中选择的任意一个或多个元素。
扫地机器人是家庭智能终端重要的一环，智能化的扫地机器人需要对自身的位置进行定位的同时，还需要感知周边的物体信息以进行路径规划，避开障碍物，例如鞋子、椅子脚、袜子等，实现高效的清扫过程。因此，障碍物检测分析是智能扫地机路径规划中重要的技术组成部分。
扫地机器人一般有两种类型：1)基于单线激光雷达的扫地机器人；2)基于双目或者单目摄像头的扫地机器人。对于单线激光雷达的方案，其优点是覆盖的角度范围大，一般的单线激光雷达都可以达到实时360度的扫描范围。然而，单线激光雷达只能对单一水平面进行扫描，因此，对于不同高度的障碍物无法完全覆盖。例如袜子紧贴地面时无法被检测。类似激光雷达，使用多线激光雷达可以一定程度上缓解这种问题，但目前多线激光雷达成本高，不适合应用在家庭扫地机这类消费级产品上。也有一些基于红外测距传感器的方案，但这类方案都有可检测范围稀疏的问题。对于第二种类型，基于双目或者单目视觉的方案，这类方案由于可以获得图像，避免了单线激光雷达返回结果稀疏的问题，同时成本也不高。目前基于视 觉的方案有两种策略，一种策略是通过目标检测，检测出障碍物在图像中的位置，使用一个矩形框标注出来，然后根据相机的参数把障碍物位置从图像坐标投影到世界坐标。这种基于目标检测的方法，可以获得一定的语义信息，例如使用一个袜子检测器，可以检测出袜子，使用拖鞋检测器，可以检测出拖鞋。然而，家庭场景内还存在很多形状多变的物体，例如电线，桌子腿的一部分等，这些障碍物很难用一个矩形框紧致地框出来。即使用一个过大的矩形框框住，忽略了障碍物的形状信息，会使得后续扫地机器人避障的时候无法紧贴障碍物行驶。基于视觉的方案另外的一种策略是使用双目相机得到前方场景的深度图信息，然后通过深度信息进行障碍物分析实现避障。然而这种方法对双目相机两个相机传感器之间有一定的位置约束(例如两个相机必须间隔一定的距离，且水平向上所处高度相等)，而且该方案对于纹理比较小，例如白墙的场景适应性不好。
针对相关的基于视觉的障碍物分析方案中，无法描述障碍物形状不固定的情况、以及对白墙场景适应性不好等问题，本公开实施例提出使用单目相机的方案，融合目标检测技术和语义分割技术的障碍物分析方法。
因此，针对扫地机器人场景下的障碍物分析存在的问题，本公开实施例提出一种可以应用于家居场所扫地机器人实现的障碍检测方法。在一些可能的实施方法，本公开实施例提出的障碍检测方法也可以应用于其他机器人，如清洁机器人、割草机器人、户外送货机器人等，在此不再赘述。
请参阅图1，图1是本公开实施例提供的障碍检测方法一实施例的流程示意图。本公开实施例的障碍检测方法的执行主体可以是一种障碍检测装置，例如，障碍检测方法可以由终端设备或服务器或其它处理设备执行，其中，障碍检测装置可以为用户设备(User Equipment，UE)、移动设备、用户终端、终端、蜂窝电话、无线电话、个人数字处理(Personal Digital Assistant，PDA)、手持设备、计算设备、车载设备、可穿戴设备等。在一些可能的实现方式中，该障碍检测方法可以通过处理器调用存储器中存储的计算机可读指令的方式来实现。
具体而言，本公开实施例的障碍检测方法可以包括以下步骤：
步骤S11：获取待检测图像。
本公开实施例中，障碍检测装置获取机器人所处环境的待检测图像。待检测图像可以由机器人上搭载的相机部分获取，相机部分具体可以为单目红绿蓝(RGB)相机。相机模型固定在机器人的固定位置，一般设置在机器人运行方向的正面位置，能够获取到机器人前行方向上的环境图像，即相机视野需要覆盖地面以及需要分析的可能存在的障碍物。
步骤S12：将待检测图像进行物体检测，获取待检测图像中物体的位置信息。
在本公开实施例中，障碍检测装置对待检测图像进行物体检测，获取 待检测图像中物体的位置信息。对于机器人而言，待检测图像中的物体即障碍物；通过物体的位置信息，障碍检测装置可以生成有效的障碍区域。
在一些实施方式中，从形状的角度区分，本公开实施例的物体可以区分为特定物体和非特定物体。其中，特定物体是由工作人员提前预设的具有相对固定形状的物体，例如拖鞋、纸团、易拉罐等；非特定物体是形状多变的物体，例如桌子腿、电线等。
步骤S13：将待检测图像进行语义分割，获取待检测图像中地面区域的位置信息。
在本公开实施例中，障碍检测装置通过语义分割部分对待检测图像进行地面检测。其中，本步骤的语义分割部分与上述检测非特定物体的语义分割部分可以为同一部分，也可以为不同部分。
语义分割部分输入的是基于标定的畸变参数矫正后的待检测图像，输出的是待检测图像中标记为地面的若干像素点，以及若干像素点组成的地面区域。
步骤S14：基于物体的位置信息以及地面的位置信息获取障碍区域。
这样，通过障碍区域能够确定避开障碍区域的规划路径。
在本公开实施例中，障碍检测装置通过融合部分获取上述地面信息和物体信息。融合部分输入信息包括：(1)语义分割部分获取的信息，即相对于相机，当前区域中属于地面的区域以及非特定物体的位置和形状；(2)物体检测部分获取的信息，即相对于相机，运行前方特定物体的位置和类别。
由于上述地面信息和物体信息均属于相机空间中的信息，障碍检测装置可以根据上述地面信息和物体信息形成当前规划地图，其中，当前规划地图可以为二维地图或三维地图。
在本公开实施例中，由于基于物体的位置信息以及地面区域的位置信息，共同确定障碍区域，而待检测图像中地面区域的位置信息是基于语义分割获取的，使得能够提高确定的地面区域的准确性，进而能够获取到准确度高的障碍区域，进而能够有效地对环境进行障碍物分析。
在获取待检测图像的过程中，障碍检测装置还可以对待检测图像进行预处理，以提高待检测图像的精确度，具体预处理过程请参阅图2，图2是图1所示障碍检测方法中步骤S11一实施例的流程示意图。如图2所示，步骤S11具体包括以下步骤：
步骤S21：获取相机的标定参数，其中，标定参数包括畸变参数。
在本公开实施例中，在固定相机部分在机器人上的位置后，障碍检测装置需要对相机进行标定。在一些实施方式中，障碍检测装置可以通过张正友标定法标定相机部分的内参和外参。其中，内参可以包括但不限于相机的焦距以及畸变参数，外参可以包括但不限于相机的单应矩阵。畸变参 数具体包括切向畸变和径向畸变，径向畸变发生在相机坐标系转图像物理坐标系的过程中，切向畸变发生在相机制作过程，由于感光元件平面与透镜平面不平行。单应矩阵为图像物理坐标系与相机坐标系之间投影映射的变换关系。
在一些可能的实施方式中，还可以通过其他标定法标定相机部分的内参和外参，例如孟胡的平面标定方法、以及吴毅红的平行圆标定方法等。
步骤S22：获取拍摄的目标图像，并基于畸变参数对目标图像进行矫正，得到待检测图像。
在本公开实施例中，切向畸变和径向畸变会导致相机部分采集的待检测图像发生部分变形。因此，障碍检测装置获取目标图像后，需要基于畸变参数对目标图像进行校正，以减少相机部分本身由于切向畸变和径向畸变导致的图像畸变的影响，有利于提高后续路径规划的准确性。
因此，通过畸变参数矫正目标图像，得到待检测图像，有利于提高路径规划的准确性。
对于特定物体，障碍检测装置可以直接通过物体检测部分进行检测，具体检测过程请参阅图4，图4是图1所示障碍检测方法中步骤S12一实施例的流程示意图。在本实施例中，标定参数还包括变换参数，如图4所示，步骤S12具体包括以下步骤：
步骤S31：将待检测图像输入第三深度神经网络，获取待检测图像中特定物体的图像位置信息。
在本公开实施例中，物体检测部分包括一个第三深度神经网络和一个几何投影部分。第三深度神经网络输入的是基于标定的畸变参数矫正后的待检测图像，输出的是包含特定物体的图像位置信息的待检测图像。其中，特定物体的图像位置信息通过图3中的物体框体现。在图3示出了特定物体包括两个纸团和一个易拉罐。
在一些实施方式中，步骤S31可以通过以下方式实现：将待检测图像输入第三深度神经网络，获取待检测图像中特定物体的包围框；基于包围框的对角坐标计算特定物体的图像位置信息。
在一些实施方式中，第三深度神经网络预先对本公开实施例中的特定物体进行训练，第三深度神经网络可以识别出特定物体的位置和类别，通过物体框的形式体现。物体框可以由围绕特定物体的包围框以及语义标识组成，语义标识即标明对应包围框中的特定物体的类别。
因此，提供通过包围框的对角坐标计算特定物体的位置信息的计算方法。
步骤S32：基于变换参数，将物体的图像位置信息变换为特定物体在相机坐标系的位置信息。
在本公开实施例中，障碍检测装置基于变换参数将物体的图像位置信 息变换为特定物体在相机坐标系的位置信息，并通过包围框的方式体现。其中，变换参数为标定的相机部分的外参，包括但不限于相机的单应矩阵。在一些实施方式中，障碍检测装置通过几何投影部分以包围框的左上角坐标和右下角坐标之间的中点位置，或者右上角坐标和左下角坐标之间的重点位置作为特定物体在图像中的位置，然后结合预先标定好的单应矩阵计算出特定物体相对相机的位置。
因此，通过第一深度神经网络的语义标签将待检测图像中的地面区域和背景区域区分开，进一步将待检测图像的信息通过变换参数投影到相机坐标系，有利于体现地面区域、背景区域与相机的距离关系。
在一些实施方式中，在得到特定物体在相机坐标系的包围框后，障碍检测装置将特定物体的类别信息标注在包围框上，以形成图3所示的特定物体的物体框。在后续的路径规划过程中，考虑特定物体的类别信息有利于规划出更加贴近真实生活，具有较高实用性的规划路径。例如，在特定物体属于鞋子等不会影响到机器人的物体的情况下，规划路径可以紧贴特定物体进行规划；在特定物体属于台灯、电热器等较危险的物体的情况下，规划路径可以与特定物体保持一定距离进行规划，防止机器人运行过程中引发危险。
对于非特定物体，障碍检测装置可以通过语义分割的方式进行检测，值得说明的是，通过语义分割方式检测非特定物体的过程，可以是通过上述的物体检测部分进行检测，也可以是通过另一语义分割部分进行检测，具体检测过程请参阅图5，图5是图1所示障碍检测方法中步骤S12另一实施例的流程示意图。在本实施例中，标定参数还包括变换参数，待检测图像中物体包括非特定物体，如图5所示，步骤S12具体包括以下步骤：
步骤S41：将待检测图像输入第二深度神经网络，获取待检测图像中非特定物体的轮廓框。
在本公开实施例中，第二深度神经网络输入的是基于标定的畸变参数矫正后的待检测图像，输出的是包含非特定物体的图像位置信息的待检测图像。其中，非特定物体的图像位置信息通过围绕非特定物体的轮廓框体现。
步骤S42：基于变换参数，将非特定物体的轮廓框变换为在相机坐标系的轮廓框，得到非特定物体在相机坐标系的位置信息。
在本公开实施例中，障碍检测装置基于单应矩阵将非特定物体的图像位置信息变换为非特定物体在相机坐标系的位置信息，并通过轮廓框的方式体现。
上述第三深度神经网络只能输出特定物体的物体框，其中，物体框为包围特定物体的矩形框；第二深度神经网络的识别原理是：对非特定物体的边缘点进行识别，然后将若干个边缘点进行组合连接，从而形成一条闭 合的边缘线，即围绕非特定物体的轮廓框。相较于特定物体的矩形物体框，非特定物体的轮廓框能够更好地体现物体的具体形状信息，有利于机器人的运行路径能够紧贴非特定物体进行规划，以提高机器人规划路径的实用性。
因此，通过第二深度神经网络对待检测图像中的非特定物体进行语义分割，以标定非特定物体的轮廓框，通过轮廓框表示非特定物体的形状，有利于在路径规划中紧贴非特定物体得到实用性高的规划路径。
对待检测图像进行地面与背景的语义分割具体操作过程请参阅图6，图6是图1所示障碍检测方法中步骤S13一实施例的流程示意图。在本实施例中，标定参数还包括变换参数，如图6所示，步骤S13具体包括以下步骤：
步骤S51：将待检测图像输入第一深度神经网络，获取待检测图像中所有像素点的语义标签，其中，语义标签包括地面标签和背景标签。
在本公开实施例中，语义分割部分包括一个全卷积的第一深度神经网络和一个几何投影部分。第一深度神经络输入的是基于标定的畸变参数矫正后的待检测图像，输出的是待检测图像中每一个像素点的语义标签。
需要说明的是，语义分割部分同样可以用于分割出非特定物体的轮廓框，即上述第二深度神经网络与本步骤的第一深度神经网络可以为同一深度神经网络，在此不再赘述。
在一些实施方式中，语义标签具体包括地面标签和背景标签。在某一像素点识别为地面像素点的情况下，语义标签标记为1；在某一像素点识别为背景像素点的情况下，语义标签标记为0。
步骤S52：基于变换参数，将待检测图像中地面标签对应的像素点变换为在相机坐标系的地面区域，得到地面在相机坐标系的位置信息。
在本公开实施例中，障碍检测装置基于单应矩阵将每个语义标签为1的地面像素点从图像空间投影到相机空间，获取地面像素点在相机坐标系的位置信息。然后，障碍检测装置将投影后的地面像素点组合成为相机坐标系的地面区域，其余区域为背景区域。其中，地面区域为机器人的行驶区域。
上述方案，障碍检测装置获取待检测图像；将待检测图像进行物体检测，获取待检测图像中物体的位置信息；将待检测图像进行语义分割，获取待检测图像中地面的位置信息；基于物体的位置信息以及地面区域的位置信息获取障碍区域。上述方案中，障碍检测装置通过对待检测图像进行物体检测以及语义分割，能够自动识别并标注出待检测图像中的地面区域和物体位置，其中，地面区域为机器人的行驶区域，而检测出的物体位置出现在行驶区域中，即可分析出该物体为障碍物，从而能够有效地对环境进行障碍物分析，并进一步根据分析结果进行路径规划。
本领域技术人员可以理解，在具体实施方式的上述方法中，各步骤的 撰写顺序并不意味着严格的执行顺序而对实施过程构成任何限定，各步骤的具体执行顺序应当以其功能和可能的内在逻辑确定。
因此，通过第一深度神经网络的语义标签将待检测图像中的地面区域和背景区域区分开，进一步将待检测图像的信息通过变换参数投影到相机坐标系，有利于体现地面区域、背景区域与相机的距离关系。
在一些实施例中，障碍检测方法还可以包括：基于物体的位置信息以及地面的位置信息形成当前规划地图，其中，当前规划地图包括可行驶区域，以及包括物体的障碍区域；基于当前规划地图的可行驶区域获取规划路径。
因此，通过地面的位置信息获得行驶区域，再通过物体的位置信息获得行驶区域中的障碍区域，从而生成当前规划地图，用于进行路径规划。
当前规划地图包括可行驶区域和物体的障碍区域。障碍区域是根据物体的位置信息所生成的区域，可行驶区域在为地面区域中除去障碍区域的部分区域。在一些实施例中，在待检测图像中包括非特定物体的情况下，非特定物体在相机坐标系的轮廓框所围成的区域可认为非特定物体的所在，因此该区域即为障碍区域(或障碍区域的一部分)；在待检测图像中包括特定物体的情况下，可通过上述的第三深度神经网络(或其它方式)获取到特定物体的类别信息，然后根据特定物体的位置信息和类别信息生成该特定物体在相机坐标系的物体框，而特定物体在相机坐标系的物体框所围成的区域可认为特定物体的所在，因此该区域即为障碍区域(或障碍区域的一部分)。值得说明的是，对于不同的特定物体，其所对应的物体框大小可能会存在区别，这是考虑到不同的特定物体对于机器人的影响不同，因此会对应不同大小的物体框以便机器人规划出更精确的路径；例如在特定物体属于鞋子等不会影响到机器人的物体的情况下，物体框的尺寸较小，也即障碍区域较小，规划路径可以紧贴特定物体进行规划；在特定物体属于台灯、电热器等较危险的物体的情况下，物体框的尺寸较大，也即障碍区域较大，规划路径可以与特定物体保持一定距离进行规划，防止机器人运行过程中引发危险。在后续的路径规划过程中，考虑特定物体的类别信息有利于规划出更加贴近真实生活，具有较高实用性的规划路径。
通过排除障碍区域，障碍检测装置可以得到行驶区域中机器人的可行驶区域，然后将可行驶区域输入轨迹规划部分。障碍检测装置通过轨迹规划部分输入终点信息，获取避开障碍区域的规划路径。
在一些实施例中，障碍检测方法还可以包括：响应于待检测图像中物体包括非特定物体的情况，确定障碍区域包括非特定物体在相机坐标系的轮廓框所对应的区域。
在另一些实施例中，障碍检测方法还可以包括：响应于待检测图像中 物体包括特定物体的情况，获取特定物体的类别信息；根据特定物体的位置信息和类别信息生成特定物体在相机坐标系的物体框；其中，障碍区域包括特定物体在相机坐标系的物体框所对应的区域。
因此，提出一种形成障碍区域的方法，有利于根据物体框快速生成行驶区域中的障碍区域；而对于特定物体，考虑特定物体的类别信息能够使障碍检测装置决定是否需要紧贴特定物体进行路径规划，有利于提高规划路径的实用性。
请参阅图7，图7是本公开实施例提供的障碍检测装置一实施例的框架示意图。障碍检测装置70包括：
相机部分71，被配置为获取待检测图像。
物体检测部分72，被配置为将待检测图像进行物体检测，获取待检测图像中物体的位置信息。
语义分割部分73，被配置为将待检测图像进行语义分割，获取待检测图像中地面区域的位置信息。
障碍检测部分74，被配置为基于物体的位置信息以及地面区域的位置信息获取障碍区域。
在一些实施例中，相机部分71，被配置为获取相机的标定参数，其中，标定参数包括畸变参数；获取拍摄的目标图像，并基于畸变参数对目标图像进行矫正，得到待检测图像。
在一些实施例中，标定参数还包括变换参数；语义分割部分73，被配置为将待检测图像输入第一深度神经网络，获取待检测图像中所有像素点的语义标签，其中，语义标签包括地面标签和背景标签；基于变换参数，将待检测图像中地面标签对应的像素点变换为在相机坐标系的地面区域，得到地面区域在相机坐标系的位置信息。
在一些实施例中，标定参数还包括变换参数，待检测图像中物体包括非特定物体；物体检测部分72，还被配置为将待检测图像输入第二深度神经网络，获取待检测图像中非特定物体的轮廓框；基于变换参数，将非特定物体的轮廓框变换为在相机坐标系的轮廓框，得到非特定物体在相机坐标系的位置信息。
在一些实施例中，标定参数还包括变换参数，待检测图像中物体包括特定物体；物体检测部分72，还被配置为将待检测图像输入第三深度神经网络，获取待检测图像中特定物体的图像位置信息；基于变换参数，将物体的图像位置信息变换为特定物体在相机坐标系的位置信息。
在一些实施例中，物体检测部分72，还被配置为将待检测图像输入第三深度神经网络，获取待检测图像中特定物体的包围框；基于包围框的对角坐标计算特定物体的图像位置信息。
在一些实施例中，装置还包括：路径规划部分，被配置为基于物体的 位置信息以及地面的位置信息形成当前规划地图，其中，当前规划地图包括可行驶区域，以及包括物体的障碍区域；基于当前规划地图的可行驶区域获取规划路径。
在一些实施例中，装置还包括：区域确定部分，被配置为响应于待检测图像中物体包括非特定物体的情况，确定障碍区域包括非特定物体在相机坐标系的轮廓框所对应的区域；或者，被配置为响应于待检测图像中物体包括特定物体的情况，获取特定物体的类别信息；根据特定物体的位置信息和类别信息生成特定物体在相机坐标系的物体框；其中，障碍区域包括特定物体在相机坐标系的物体框所对应的区域。
在本公开实施例中，用于实现本公开实施例的方案的系统可以包括以下部分：相机部分、物体检测部分、地面分割部分(对应上述实施例中的语义分割部分)、融合部分(对应上述实施例中的障碍检测部分)以及轨迹规划部分。
在实施过程中，需要定义一些有相对固定形状的物体，例如袜子，鞋子，这里先把这些物体称为集合A。
系统实际部署的时候，相机部分可以接受到当前的图片，把图片分别分发到物体检测部分和语义分割部分。物体检测部分检测出图片中所有集合A里面的物体，输出物体类别以及物体相对于当前机器人的位置。而地面分割部分则针对输入的图片，对图片中的像素打标签，输出属于平地的区域。结合平地区域以及物体集合A的位置，融合部分输出扫地机器人可行驶的区域地图。根据这个地图，轨迹规划部分规划出避障的轨迹，最后系统控制部分完成避障轨迹的执行。
相机部分需固定在扫地机器人某个固定的位置，以确定相机部分的视野能否覆盖地面以及需要分析的可能存在的障碍物。固定相机部分后，需要有一个标定的过程。一方面标定相机部分的内参I(包括焦距以及畸变参数)，另一方面标定外参E(这里外参指的是相机成像平面与地面两个平面之间的单应矩阵)。这里的标定可以采用张正友标定法。
物体检测部分的输入可以是基于标定的畸变参数矫正后的图像，输出是特定类型的障碍物的类别和位置。这里，特定类型的障碍物是由系统开发者开发前制定的具有相对固定形状的物体(例如拖鞋、纸团、易拉罐等)。具体来说，物体检测部分可以包括一个深度神经网络M1和一个几何投影部分G。神经网络A输入的是图片，输出是检测到的物体的包围框，然后，几何投影部分以包围框的左下角和右下角两点之间的中点作为物体在图像中的位置，结合预先标定好的外参E，可以算出物体相对相机部分的位置。
地面分割部分的输入是基于标定的畸变参数矫正后的图像，输出的是输入图像中每个像素的语义标签。具体来讲，语义标签等于0或者1，等于1的时候表示该像素为地面，否则为背景。地面分割部分可以由一个全卷积 的深度网络M2实现。得到每个像素的语义标签后，通过几何部分G，可以把每个语义为地面的像素从图像空间投影到相机空间(即相对于相机的坐标)。
融合部分可以确定以下信息：1)地面分割部分得到的信息，即相对于相机部分，当前哪些区域是地面；2)物体检测部分得到的信息，即相对于相机部分，前方在什么位置有什么障碍物体。这两方面的信息都是在相机空间系里面的，所以综合分析这两方面的信息，融合部分得到当前相机部分覆盖的范围内，有哪些区域是可行驶的。可行驶的区域可以使用一个局部地图来描述。将局部地图输入到轨迹规划部分，即可以得到避障的轨迹。
在本公开实施例中，通过使用单目RGB相机完成扫地机通用障碍物规避功能，通过融合物体检测和语义分割技术实现单目RGB的场景障碍物分析功能，不仅可以降低扫地机器避障的硬件成本要求，还可以对特定类型以及非特定类型的障碍物进行检测分析，提高扫地机器人的避障效果。
在本公开实施例中，在自主导航的移动机器人上，使用本方案进行障碍物分析，从而得知机器人前面有哪些障碍物以及哪些是可以行走的区域，根据这些信息反馈给规划控制系统，从而实现障碍物规避。
请参阅图8，图8是本公开实施例提供的电子设备一实施例的框架示意图。电子设备80包括相互耦接的存储器81和处理器82，处理器82被配置为执行存储器81中存储的程序指令，以实现上述任一障碍检测方法实施例中的步骤。在一个实施场景中，电子设备80可以包括但不限于：微型计算机、服务器，此外，电子设备80还可以包括笔记本电脑、平板电脑等移动设备，在此不做限定。
具体而言，处理器82被配置为控制其自身以及存储器81以实现上述任一障碍检测方法实施例中的步骤。处理器82还可以称为中央处理单元(Central Processing Unit，CPU)。处理器82可能是一种集成电路芯片，具有信号的处理能力。处理器82还可以是通用处理器、数字信号处理器(Digital Signal Processor，DSP)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现场可编程门阵列(Field-Programmable Gate Array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。另外，处理器82可以由集成电路芯片共同实现。
请参阅图9，图9是本公开实施例提供的计算机可读存储介质一实施例的框架示意图。计算机可读存储介质90存储有能够被处理器运行的程序指令901，程序指令901用于实现上述任一障碍检测方法实施例中的步骤。
在一些实施例中，本公开实施例提供的装置具有的功能或包含的部分可以用于执行上文方法实施例描述的方法，其具体实现可以参照上文方法实施例的描述，为了简洁，这里不再赘述。
本公开实施例还可以提供一种计算机程序，包括计算机可读代码，当所述计算机可读代码在电子设备中运行时，所述电子设备中的处理器执行用于实现上述任一项所述的障碍检测方法实施例中的步骤。
上文对各个实施例的描述倾向于强调各个实施例之间的不同之处，其相同或相似之处可以互相参考，为了简洁，本文不再赘述。
在本公开实施例所提供的几个实施例中，应该理解到，所揭露的方法和装置，可以通过其它的方式实现。例如，以上所描述的装置实施方式仅仅是示意性的，例如，部分或单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性、机械或其它的形式。
另外，在本公开实施例各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本公开实施例的技术方案本质上或者说对相关技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)或处理器(processor)执行本公开实施例各个实施方式方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质。
工业实用性
本公开实施例中，由于基于所述物体的位置信息以及所述地面区域的位置信息，共同确定障碍区域，而待检测图像中地面区域的位置信息是基于语义分割获取的，使得能够提高确定的地面区域的准确性，进而能够获取到准确度高的障碍区域，从而能够有效地对环境进行障碍物分析。

Claims (19)

  1. 一种障碍检测方法，所述障碍检测方法包括：
    获取待检测图像；
    将所述待检测图像进行物体检测，获取所述待检测图像中物体的位置信息；
    将所述待检测图像进行语义分割，获取所述待检测图像中地面区域的位置信息；
    基于所述物体的位置信息以及所述地面区域的位置信息获取障碍区域。
  2. 根据权利要求1所述的障碍检测方法，其中，
    所述获取待检测图像，包括：
    获取相机的标定参数，其中，所述标定参数包括畸变参数；
    获取拍摄的目标图像，并基于所述畸变参数对所述目标图像进行矫正，得到所述待检测图像。
  3. 根据权利要求2所述的障碍检测方法，其中，所述标定参数还包括变换参数；
    所述将所述待检测图像进行语义分割，获取所述待检测图像中地面区域的位置信息，包括：
    将所述待检测图像输入第一深度神经网络，获取所述待检测图像中所有像素点的语义标签，其中，所述语义标签包括地面标签和背景标签；
    基于所述变换参数，将所述待检测图像中所述地面标签对应的像素点变换为在所述相机坐标系的地面区域，得到所述地面区域在所述相机坐标系的位置信息。
  4. 根据权利要求2或3所述的障碍检测方法，其中，所述标定参数还包括变换参数，所述待检测图像中物体包括非特定物体；
    所述将所述待检测图像进行物体检测，获取所述待检测图像中物体的位置信息，还包括：
    将所述待检测图像输入第二深度神经网络，获取所述待检测图像中所述非特定物体的轮廓框；
    基于所述变换参数，将所述非特定物体的轮廓框变换为在所述相机坐标系的轮廓框，得到所述非特定物体在所述相机坐标系的位置信息。
  5. 根据权利要求2至4任一项所述的障碍检测方法，其中，所述标定参数还包括变换参数，所述待检测图像中物体包括特定物体；
    所述将所述待检测图像进行物体检测，获取所述待检测图像中物体的位置信息，包括：
    将所述待检测图像输入第三深度神经网络，获取所述待检测图像中特 定物体的图像位置信息；
    基于所述变换参数，将所述物体的图像位置信息变换为所述特定物体在所述相机坐标系的位置信息。
  6. 根据权利要求5所述的障碍检测方法，其中，
    所述将所述待检测图像输入第三深度神经网络，获取所述待检测图像中特定物体的图像位置信息，包括：
    将所述待检测图像输入所述第三深度神经网络，获取所述待检测图像中特定物体的包围框；
    基于所述包围框的对角坐标计算所述特定物体的图像位置信息。
  7. 根据权利要求1至6中任一项所述的障碍检测方法，其中，
    所述障碍检测方法还包括：
    基于所述物体的位置信息以及所述地面的位置信息形成当前规划地图，其中，所述当前规划地图包括可行驶区域，以及包括所述物体的障碍区域；
    基于所述当前规划地图的可行驶区域获取规划路径。
  8. 根据权利要求7所述的障碍检测方法，其中，所述障碍检测方法还包括：
    响应于所述待检测图像中物体包括非特定物体的情况，确定所述障碍区域包括所述非特定物体在所述相机坐标系的轮廓框所对应的区域；和/或，
    响应于所述待检测图像中物体包括特定物体的情况，获取所述特定物体的类别信息；根据所述特定物体的位置信息和类别信息生成所述特定物体在相机坐标系的物体框；其中，所述障碍区域包括所述特定物体在所述相机坐标系的物体框所对应的区域。
  9. 一种障碍检测装置，所述障碍检测装置包括：
    相机部分，被配置为获取待检测图像；
    物体检测部分，被配置为将所述待检测图像进行物体检测，获取所述待检测图像中物体的位置信息；
    语义分割部分，被配置为将所述待检测图像进行语义分割，获取所述待检测图像中地面区域的位置信息；
    障碍检测部分，被配置为基于所述物体的位置信息以及所述地面区域的位置信息获取障碍区域。
  10. 根据权利要求9所述的装置，其中，所述相机部分，被配置为获取相机的标定参数，其中，所述标定参数包括畸变参数；获取拍摄的目标图像，并基于所述畸变参数对所述目标图像进行矫正，得到所述待检测图像。
  11. 根据权利要求10所述的装置，其中，所述标定参数还包括变换参数；所述语义分割部分，被配置为将所述待检测图像输入第一深度神经网络，获取所述待检测图像中所有像素点的语义标签，其中，所述语义标签包括地面标签和背景标签；基于所述变换参数，将所述待检测图像中所述 地面标签对应的像素点变换为在所述相机坐标系的地面区域，得到所述地面区域在所述相机坐标系的位置信息。
  12. 根据权利要求10或11所述的装置，其中，所述标定参数还包括变换参数，所述待检测图像中物体包括非特定物体；所述物体检测部分，还被配置为将所述待检测图像输入第二深度神经网络，获取所述待检测图像中所述非特定物体的轮廓框；基于所述变换参数，将所述非特定物体的轮廓框变换为在所述相机坐标系的轮廓框，得到所述非特定物体在所述相机坐标系的位置信息。
  13. 根据权利要求10至12任一项所述的装置，其中，所述标定参数还包括变换参数，所述待检测图像中物体包括特定物体；所述物体检测部分，还被配置为将所述待检测图像输入第三深度神经网络，获取所述待检测图像中特定物体的图像位置信息；基于所述变换参数，将所述物体的图像位置信息变换为所述特定物体在所述相机坐标系的位置信息。
  14. 根据权利要求13所述的装置，其中，所述物体检测部分，还被配置为将所述待检测图像输入所述第三深度神经网络，获取所述待检测图像中特定物体的包围框；基于所述包围框的对角坐标计算所述特定物体的图像位置信息。
  15. 根据权利要求9至14任一项所述的装置，其中，所述装置还包括：
    路径规划部分，被配置为基于所述物体的位置信息以及所述地面的位置信息形成当前规划地图，其中，所述当前规划地图包括可行驶区域，以及包括所述物体的障碍区域；基于所述当前规划地图的可行驶区域获取规划路径。
  16. 根据权利要求15所述的装置，其中，所述装置还包括：
    区域确定部分，被配置为响应于所述待检测图像中物体包括非特定物体的情况，确定所述障碍区域包括所述非特定物体在所述相机坐标系的轮廓框所对应的区域；和/或，被配置为响应于所述待检测图像中物体包括特定物体的情况，获取所述特定物体的类别信息；根据所述特定物体的位置信息和类别信息生成所述特定物体在相机坐标系的物体框；其中，所述障碍区域包括所述特定物体在所述相机坐标系的物体框所对应的区域。
  17. 一种电子设备，包括相互耦接的存储器和处理器，所述处理器被配置为执行所述存储器中存储的程序指令，以实现权利要求1至8任一项所述的障碍检测方法。
  18. 一种计算机可读存储介质，其上存储有程序指令，所述程序指令被处理器执行时实现权利要求1至8任一项所述的障碍检测方法。
  19. 一种计算机程序，包括计算机可读代码，当所述计算机可读代码在电子设备中运行时，所述电子设备中的处理器执行用于实现如权利要求1至8任一项所述的障碍检测方法。
PCT/CN2021/085927 2020-10-28 2021-04-08 障碍检测方法、装置、电子设备、存储介质及计算机程序 WO2022088611A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
KR1020217034499A KR20220060500A (ko) 2020-10-28 2021-04-08 장애 검출 방법, 장치, 전자 기기, 저장 매체 및 컴퓨터 프로그램
JP2021562323A JP2023503747A (ja) 2020-10-28 2021-04-08 障害検出方法及び装置、電子機器、記憶媒体並びにコンピュータプログラム

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202011172403.6A CN112183476B (zh) 2020-10-28 2020-10-28 一种障碍检测方法、装置、电子设备以及存储介质
CN202011172403.6 2020-10-28

Publications (1)

Publication Number Publication Date
WO2022088611A1 true WO2022088611A1 (zh) 2022-05-05

Family

ID=73923531

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/085927 WO2022088611A1 (zh) 2020-10-28 2021-04-08 障碍检测方法、装置、电子设备、存储介质及计算机程序

Country Status (4)

Country Link
JP (1) JP2023503747A (zh)
KR (1) KR20220060500A (zh)
CN (1) CN112183476B (zh)
WO (1) WO2022088611A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115376109A (zh) * 2022-10-25 2022-11-22 杭州华橙软件技术有限公司 障碍物检测方法、障碍物检测装置以及存储介质

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112183476B (zh) * 2020-10-28 2022-12-23 深圳市商汤科技有限公司 一种障碍检测方法、装置、电子设备以及存储介质
CN112711263A (zh) * 2021-01-19 2021-04-27 未来机器人（深圳）有限公司 仓储自动导引车避障方法、装置、计算机设备和存储介质
CN113158779B (zh) * 2021-03-09 2024-06-07 美智纵横科技有限责任公司 一种行走方法、装置和计算机存储介质
CN112966658B (zh) * 2021-03-30 2024-06-18 深圳市优必选科技股份有限公司 机器人导航方法、装置、终端设备及计算机可读存储介质

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101690511B1 (ko) * 2015-09-14 2016-12-28 주식회사 서연전자 자율 주행 차량의 제어 시스템 및 그 제어 방법
CN111126251A (zh) * 2019-12-20 2020-05-08 深圳市商汤科技有限公司 图像处理方法、装置、设备和存储介质
CN112183476A (zh) * 2020-10-28 2021-01-05 深圳市商汤科技有限公司 一种障碍检测方法、装置、电子设备以及存储介质

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5635770B2 (ja) * 2009-12-24 2014-12-03 京セラ株式会社 障害物検出装置
CN105115490A (zh) * 2015-07-16 2015-12-02 深圳前海达闼科技有限公司 一种确定室内可活动区域的方法及装置
CN110246142A (zh) * 2019-06-14 2019-09-17 深圳前海达闼云端智能科技有限公司 一种检测障碍物的方法、终端和可读存储介质
CN110502019A (zh) * 2019-09-06 2019-11-26 北京云迹科技有限公司 一种室内机器人的避障方法及装置
CN110974088B (zh) * 2019-11-29 2021-09-24 深圳市杉川机器人有限公司 扫地机器人控制方法、扫地机器人及存储介质
CN111462135B (zh) * 2020-03-31 2023-04-21 华东理工大学 基于视觉slam与二维语义分割的语义建图方法

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101690511B1 (ko) * 2015-09-14 2016-12-28 주식회사 서연전자 자율 주행 차량의 제어 시스템 및 그 제어 방법
CN111126251A (zh) * 2019-12-20 2020-05-08 深圳市商汤科技有限公司 图像处理方法、装置、设备和存储介质
CN112183476A (zh) * 2020-10-28 2021-01-05 深圳市商汤科技有限公司 一种障碍检测方法、装置、电子设备以及存储介质

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115376109A (zh) * 2022-10-25 2022-11-22 杭州华橙软件技术有限公司 障碍物检测方法、障碍物检测装置以及存储介质

Also Published As

Publication number Publication date
KR20220060500A (ko) 2022-05-11
CN112183476B (zh) 2022-12-23
JP2023503747A (ja) 2023-02-01
CN112183476A (zh) 2021-01-05

Similar Documents

Publication Publication Date Title
WO2022088611A1 (zh) 障碍检测方法、装置、电子设备、存储介质及计算机程序
US10740920B1 (en) Method and apparatus for combining data to construct a floor plan
Kang et al. Automatic targetless camera–lidar calibration by aligning edge with gaussian mixture model
CN111126251B (zh) 图像处理方法、装置、设备和存储介质
Menze et al. Object scene flow for autonomous vehicles
WO2020102944A1 (zh) 点云处理方法、设备及存储介质
WO2019120011A1 (zh) 一种目标检测方法及装置
Liang et al. Image based localization in indoor environments
Treible et al. Cats: A color and thermal stereo benchmark
Wang et al. Color-based road detection and its evaluation on the KITTI road benchmark
EP4283567A1 (en) Three-dimensional map construction method and apparatus
US9886769B1 (en) Use of 3D depth map with low and high resolution 2D images for gesture recognition and object tracking systems
WO2022078513A1 (zh) 定位方法、装置、自移动设备和存储介质
CN112700552A (zh) 三维物体检测方法、装置、电子设备及介质
CN110838144B (zh) 一种充电设备识别方法、移动机器人和充电设备识别系统
CN113359692B (zh) 一种障碍物的避让方法、可移动机器人
WO2022078488A1 (zh) 定位方法、装置、自移动设备和存储介质
WO2021114776A1 (en) Object detection method, object detection device, terminal device, and medium
Gu et al. Histograms of the normalized inverse depth and line scanning for urban road detection
CN112883955A (zh) 货架布局检测方法、装置及计算机可读存储介质
TW202029134A (zh) 行車偵測方法、車輛及行車處理裝置
Cociaş et al. Multiple-superquadrics based object surface estimation for grasping in service robotics
Cheng et al. Structured light-based shape measurement system
CN114155349A (zh) 一种三维建图方法、三维建图装置及机器人
Bodensteiner et al. Accurate single image multi-modal camera pose estimation

Legal Events

Date Code Title Description
ENP Entry into the national phase

Ref document number: 2021562323

Country of ref document: JP

Kind code of ref document: A

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21884355

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 17.08.2023)

122 Ep: pct application non-entry in european phase

Ref document number: 21884355

Country of ref document: EP

Kind code of ref document: A1