WO2022227939A1 - 地面障碍物检测方法以及使用该方法的移动机器 - Google Patents

地面障碍物检测方法以及使用该方法的移动机器 Download PDF

Info

Publication number
WO2022227939A1
WO2022227939A1 PCT/CN2022/082148 CN2022082148W WO2022227939A1 WO 2022227939 A1 WO2022227939 A1 WO 2022227939A1 CN 2022082148 W CN2022082148 W CN 2022082148W WO 2022227939 A1 WO2022227939 A1 WO 2022227939A1
Authority
WO
WIPO (PCT)
Prior art keywords
channel data
obstacle
rgb
image
depth channel
Prior art date
Application number
PCT/CN2022/082148
Other languages
English (en)
French (fr)
Inventor
邵丹
郭德骏
修震
董初桥
谭欢
Original Assignee
深圳市优必选科技股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳市优必选科技股份有限公司 filed Critical 深圳市优必选科技股份有限公司
Priority to CN202280003657.6A priority Critical patent/CN115516523A/zh
Publication of WO2022227939A1 publication Critical patent/WO2022227939A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/02—Control of position or course in two dimensions
    • G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0231—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means
    • G05D1/0246—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means
    • G05D1/0251—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/803—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of input or preprocessed data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/64—Three-dimensional objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30252—Vehicle exterior; Vicinity of vehicle
    • G06T2207/30261—Obstacle

Definitions

  • the invention relates to obstacle detection technology, in particular to a ground obstacle detection method and a mobile machine using the method.
  • AI artificial intelligence
  • mobile machines such as mobile robots and vehicles are being widely used. Since mobile machines usually have navigation capabilities to perform tasks such as movement and transportation, obstacle avoidance needs to be implemented to prevent them from being affected by obstacles, thereby ensuring the execution of tasks.
  • sensors are often limited by the blind spot problem, where the sensor cannot distinguish small (or micro) obstacles from the presence of the ground, and instead reports the presence of all objects including obstacles and the ground. Therefore, they are installed in front of mobile machines and only sense obstacles and not the ground. That is, these sensors will not be able to cover small (or micro) ground obstacles below their mounting height.
  • an RGB-D sensor (or camera) is economical and can be mounted on a mobile machine to detect the ground ahead and ensure that its field of view covers every place the mobile machine travels. Therefore, it is possible to use RGB-D sensors to achieve better detection of tiny and small ground obstacles if an algorithm is provided that can distinguish obstacles from the background.
  • the present application provides a method for detecting obstacles on the ground and a mobile machine using the method, which are used to detect obstacles on the floor, and solve the problems existing in the above-mentioned methods for detecting obstacles on the ground in the prior art.
  • Embodiments of the present application provide an obstacle detection method for detecting obstacles on the floor using an RGB-D camera, the method comprising:
  • RGB-D camera receives an image through an RGB-D camera, wherein the image includes depth channel data and RGB channel data;
  • the 2D position of the obstacle is converted to a 3D position of the obstacle based on the depth channel data.
  • Embodiments of the present application also provide a mobile machine, including:
  • one or more memories storing one or more computer programs for execution by the one or more processors, wherein the one or more computer programs include a plurality of instructions for:
  • RGB-D camera receives an image through an RGB-D camera, wherein the image includes depth channel data and RGB channel data;
  • the 2D position of the obstacle is converted to a 3D position of the obstacle based on the depth channel data.
  • the ground obstacle detection method provided by the present application uses an RGB-D camera to obtain an image, and performs distribution modeling on the foreground of the image to obtain the position of the obstacle, thereby solving the difficult problem in the prior art. Detect problems such as micro and small obstacles on the ground.
  • FIG. 1A is a schematic diagram of navigation of a mobile machine in some embodiments of the present application.
  • FIG. 1B is a schematic block diagram illustrating the mobile machine of FIG. 1A .
  • Figure 2A is a schematic block diagram of an example of ground obstacle detection for the mobile machine of Figure IB.
  • FIG. 2B is a schematic diagram of the ground obstacle detection of FIG. 2A .
  • FIG. 3 is a schematic diagram of an image captured by an RGB-D camera of the mobile machine of FIG. 1B .
  • FIG. 4 is a schematic block diagram of an example of ground plane estimation in the example of ground obstacle detection of FIG. 2A .
  • FIG. 5 is a schematic diagram of an example of ground plane estimation and foreground acquisition in the example of ground obstacle detection of FIG. 2A.
  • FIG. 6 is a schematic block diagram of an example of foreground acquisition in the example of ground obstacle detection of FIG. 2A .
  • FIG. 7 is a schematic block diagram of an example of distribution modeling in the example of ground obstacle detection of FIG. 2A .
  • first”, “second” and “third” in this application are only for the purpose of description, and should not be understood as indicating or implying relative importance, or implying the number of technical features referred to . Thus, the features defined by “first”, “second” and “third” may explicitly or implicitly include at least one technical feature therein.
  • “plurality” means at least two, such as two, three, etc., unless expressly defined otherwise.
  • the term “mobile machine” refers to a machine that has the ability to move around in its environment, such as a mobile robot or vehicle.
  • the term “obstacle” refers to an object on the floor that may interfere with the movement of a mobile machine.
  • sensor refers to devices, modules, machines, or subsystems, such as image sensors and gyroscopes, whose purpose is to detect events or changes in their environment and send information to other electronic devices (e.g., processors).
  • the term “navigation” refers to the process of monitoring and controlling the movement of a mobile machine from one place to another.
  • the term “collision avoidance” refers to preventing or mitigating the severity of a collision during navigation.
  • FIG. 1A is a schematic diagram of navigation of a mobile machine 100 in some embodiments of the present application.
  • Mobile machines 100 eg, mobile robots and vehicles
  • a ground obstacle O on the floor F to prevent its movement from being affected such as obstruction, deceleration, tripping or slipping, or to avoid dangerous situations such as collisions and falls.
  • the mobile device 100 navigates from the origin along a path between the origin (eg, where the mobile machine 100 is located) and the destination (eg, the location of the navigation target specified by the user or the navigation/operating system of the mobile device 100 ) To reach the destination, it is necessary to avoid obstacles O on the floor F such as falling objects, garbage, and even walls.
  • a map of the environment needs to be constructed, the current location of the mobile machine 100 in the environment needs to be determined, and a path for the mobile machine 100 to move from an origin to a destination needs to be planned.
  • navigation and/or ground obstacle detection of the mobile machine 100 may be provided by the mobile machine 100 itself (eg, a control interface on the mobile machine 100 ) or a control device 200 (eg, a remote control of the mobile machine 100 ) via, for example, A request for navigation and/or ground obstacle detection of the mobile machine 100 is initiated.
  • the control device 200 may be, for example, a smartphone, a tablet computer, a notebook computer, a desktop computer, or other electronic devices.
  • FIG. 1B is a schematic block diagram illustrating the mobile machine 100 of FIG. 1A .
  • the mobile machine 100 may be a mobile robot, such as a wheeled robot or a humanoid robot, which may include a processing unit 110, a storage unit 120 and a control unit 130 that communicate through one or more communication buses or signal lines L.
  • the mobile machine 100 is only one example of a mobile device, and the mobile machine 100 may have more or less components (eg, units, subunits, and modules) than those shown above or below, and two or more may be combined components, or may have different component configurations or arrangements.
  • Processing unit 110 executes various instructions (sets) stored in storage unit 120, which may be in the form of software programs, to perform various functions of mobile machine 100 and process related data, which may include one or more processes a processor (such as a central processing unit (CPU)).
  • Storage unit 120 may include one or more memories (eg, high-speed random access memory (RAM) and non-transitory memory), one or more memory controllers, and one or more non-transitory computer-readable storage media (eg, Solid State Drive (SSD) or hard drive).
  • RAM random access memory
  • SSD Solid State Drive
  • the control unit 130 may include various controllers (eg, a camera controller, a display controller, and a physical button controller) and a peripheral interface for coupling the input/output peripherals of the mobile machine 100 to the processing unit 110 and the storage unit 120, Examples are external ports of the processing unit 110 (eg, USB), wireless communication circuits (eg, RF communication circuits), audio circuits (eg, speaker circuits), sensors (eg, gyroscopes and cameras).
  • the storage unit 120 may include a navigation module 121 for implementing functions related to navigation (and ground obstacle detection) of the mobile machine 100 (eg, map building, path planning, and ground obstacle detection), which may Stored in one or more memories (and one or more non-transitory computer readable storage media).
  • mobile machine 100 may be a vehicle, such as an automobile.
  • the navigation module 121 in the storage unit 120 of the mobile device 100 may be a software module (of the operating system of the mobile machine 100 ) having instructions In for enabling navigation of the mobile machine 100 (eg, a motor used to drive the mobile machine 100 ). M to move the instruction of the mobile machine 100), the ground obstacle detection sub-module 1211 and the collision avoidance sub-module 1212.
  • the ground obstacle detection sub-module 1211 may be a software module having an instruction I o for detecting ground obstacles on the mobile machine 100 , so as to detect obstacles on the ground.
  • the collision avoidance sub-module 1212 may be a software module having instructions Ic for implementing collision avoidance of the mobile machine 100 so as to avoid collision with obstacles.
  • the ground obstacle detection sub-module 1211 and the collision avoidance sub-module 1212 may be sub-modules independent of the instruction In of the navigation module 121 or other sub - modules, or part of the instruction In for implementing the navigation of the mobile machine 100 .
  • the ground obstacle detection sub-module 1211 may also have data related to the ground obstacle detection of the mobile machine 100 (eg, input/output data and temporary data), which may be stored in one or more memories and accessed by the processing unit 110 .
  • the ground obstacle detection sub-module 1211 may be a module separated from the navigation module 121 in the storage unit 120 , or may be a module integrated in the anti-collision sub-module 1212 .
  • the instructions In may include instructions for implementing ground obstacle detection and collision avoidance for the mobile machine 100 .
  • the navigation module 121 may be a navigation unit that communicates with the processing unit 110, the storage unit 120 and the control unit 130 through one or more communication buses or signal lines L, and may also include a navigation unit for storing instructions In,
  • One or more memories eg, high-speed random access memory (RAM) and non-transitory memory
  • one or more processors eg, a microprocessor (MPU) of the ground obstacle detection sub-module 1211 and the collision avoidance sub-module 1212 and microcontroller (MCU)
  • Io the ground obstacle detection sub-module 1211
  • MCU microcontroller
  • Mobile machine 100 may also include sensor subunit 133 .
  • the sensor sub-unit 133 may include a set of sensors and associated controllers, such as an RGB-D camera C and a gyroscope G (or an inertial measurement unit (IMU)), for detecting the environment in which it is located to achieve its navigation.
  • the RGB-D camera C may be installed on the front of the mobile device 100 to face the direction in which the mobile machine 100 moves so that its field of view V (see FIG. 2B ) can cover every place the mobile machine 100 moves.
  • the sensor sub-unit 133 communicates with the control unit 130 through one or more communication buses or signal lines L, and the one or more communication buses or signal lines may be the same as the above-mentioned one or more communication buses or signal lines L, or at least Parts are different.
  • the sensor sub-unit 133 may communicate with the navigation unit through one or more communication buses or signal lines, and the communication bus or signal line may communicate with the above-mentioned one
  • the plurality of communication buses or signal lines L are identical, or at least partially different.
  • the sensor subunit 133 may be just an abstract component for representing the logical relationship between the components of the mobile machine 100 .
  • the mobile machine 100 may also include a communication subunit 131 and an actuation subunit 132 .
  • the communication sub-unit 131 and the actuating sub-unit 132 communicate with the control unit 130 through one or more identical communication buses or signal lines, and the one or more communication buses or signal lines may be The signal lines L are the same, or at least partially different.
  • the communication subunit 131 is coupled to a communication interface of the mobile machine 100, such as a network interface 1311, an I/O interface 1312 (eg, a physical button) for the mobile machine 100 to communicate with the control device 200 through a network.
  • the actuation subunit 132 is coupled to the components/devices used to effect the movement of the mobile machine 100 to drive the motors M of the wheels and/or joints of the mobile machine 100 .
  • the communication sub-unit 131 may include a controller for the above-described communication interface of the mobile machine 100
  • the actuation sub-unit 132 may include a controller for the above-described components/devices for enabling movement of the mobile machine 100 .
  • the communication subunit 131 and/or the actuation subunit 132 may just be abstract components to represent logical relationships between the components of the mobile machine 100 .
  • the ground obstacle detection sub-module 1211, the collision avoidance sub-module 1212, the sensor sub-unit 133 and the motor M together form one (navigation) A system that enables map building, (global and local) path planning and motor drive to enable navigation of the mobile machine 100 .
  • the various components shown in Figure IB may be implemented in hardware, software, or a combination of hardware and software.
  • Two or more of the processing unit 110, storage unit 120, control unit 130, navigation module 121 and other units/subunits/modules may be implemented on a single chip or circuit. In other embodiments, at least some of them may be implemented on separate chips or circuits.
  • FIG. 2A is a schematic block diagram of an example of ground obstacle detection for the mobile machine 100 of FIG. 1B .
  • the instructions (groups) I o corresponding to the ground obstacle detection method of the mobile machine 100 are stored, for example, as the navigation module 121 in the storage unit 120 , and the stored instructions I o are executed by the processing unit 110 .
  • the ground obstacle detection method is implemented in the mobile machine 100 in the manner of the above.
  • This ground obstacle detection may be performed in response to a request for obstacle detection on the ground of the mobile machine 100, for example from the mobile machine 100 itself or the control device 200 (the navigation/operating system of the control device 200), and then may also be performed, for example, at specific time intervals (eg 1 second) and/or re-execute when the planned path is re-planned until navigation of the mobile machine 100 is complete.
  • FIG. 2B is a schematic diagram of ground obstacle detection in FIG. 2A . As shown in Fig. 2B, the field of view V of the RGB-D camera C covers the ground obstacles O and the ground F. Through the ground obstacle detection, the obstacle O will be detected, and the 3D (three-dimensional) position L 3 of the obstacle O will be acquired.
  • FIG. 3 is a schematic diagram of an image I captured by the RGB-D camera C of the mobile machine 100 of FIG. 1B .
  • Image I is an image of a scene in the environment of FIG. 1A as the mobile machine 100 navigates.
  • Image I includes depth channel data D d and RGB channel data D r .
  • the depth channel data D d includes the depth channel of the image I for representing distances to scene objects in the image I (eg, obstacles O, ground F, and walls W on the ground).
  • the RGB channel data Dr includes a red channel, a green channel and a blue channel of the image I, and is used to represent the colors constituting the image I.
  • the processing unit 110 further estimates a ground plane Pg corresponding to the floor F based on the depth channel data Dd ( block 220 of FIG. 2A).
  • the ground plane Pg is the estimated plane in the image I and represents the floor F where the obstacle O is located.
  • the ground plane P g may be estimated based on the depth channel data D d of the images I, eg, by estimating the ground plane P g of each image I, and taking the One of the estimated ground planes Pg (eg the ground plane Pg present in multiple images I).
  • FIG. 4 is a schematic block diagram of an example of ground plane estimation in the example of ground obstacle detection of FIG. 2A . In some embodiments, in the ground plane estimation (block 220 of FIG.
  • each patch represents a small area of the scene that can always approximately represent the plane Ps in the depth channel data Dd .
  • each planar point in the depth channel data D d is identified as a plane Ps .
  • the processing unit 110 further obtains the included angle ⁇ p of each plane P s relative to the RGB-D camera C (ie, the “plane-camera angle ⁇ p ” in FIG. 5 ) based on the depth channel data D d (block 222 of FIG. 4 ) . Then, it can be based on the similarity in angle and the difference in height of adjacent patch planes P s (assuming that all patch planes P s belonging to floor F have similar angles and similar heights with respect to RGB-D camera C ), the ground plane Pg is calculated by grouping the adjacent small planes Ps into one large plane .
  • the angle of each facet Ps can be calculated from the plane fitting process.
  • FIG. 5 is a schematic diagram of an example of ground plane estimation and foreground acquisition in the example of ground obstacle detection of FIG. 2A , where ground plane Pg is assumed to be the same as floor F.
  • FIG. Pg is estimated as the so-called principal plane in the field of view of camera C, the largest connected plane in the scene.
  • the RGB-D camera C when the RGB-D camera C has a greater height relative to the floor F, in order to better sense the floor F, rather than shooting forward (ie, parallel to the floor F), the RGB-D camera C may have a downward tilt Angle ⁇ c is taken at an inclination angle relative to the floor F, so the main planar structure in the field of view is the ground/floor, not a wall or other structure.
  • adjacent planes Ps with similar included angles are connected as a single plane, with The largest connecting single plane of the angle between the mean values of all connecting facets P s of the plane P g is taken as the ground plane P g .
  • the plane P s parallel to the lower surface S ol of the obstacle O is connected as a larger single plane, and the plane P s located on the upper surface S ou of the obstacle O is connected as a smaller single plane or a plurality of plane (when the curvature of the upper surface S ou is large), and one or more planes at the upper surface S ou and the plane P s at the lower surface S ol are broken due to their included angle ⁇ p or a large difference in relative height open.
  • a larger single plane connected by a plane P s parallel to the lower surface S ol of the obstacle O will be taken as the ground plane P g .
  • the processing unit 110 further obtains the foreground F i (ie, the initial foreground in FIG. 5 ) of the image I corresponding to the ground plane Pg based on the depth channel data D d (block 230 of FIG. 2A ).
  • the position of the foreground Fi (pixel in ) will be obtained.
  • the foreground F i may be obtained based on the depth channel data D d of the images I, eg, by obtaining the ground plane of each image I P g and take one (largest) foreground F i obtained.
  • FIG. 6 is a schematic block diagram of an example of foreground acquisition in the example of ground obstacle detection of FIG. 2A .
  • processing unit 110 acquires the location of each point P (see FIG. 5 ) in depth channel data D d that is at least a minimum distance from ground plane P g (i.e. each point P at a distance greater than or equal to a minimum distance (eg, 1 cm) from the ground plane Pg as a candidate location Lc for the obstacle O (block 231).
  • the candidate location Lc may include the coordinates of each point P (e.g. Cartesian coordinates (x, y, z)).
  • the processing unit 110 further uses the area in the image I corresponding to all candidate positions L c of the obstacle O as the foreground F i of the image I (block 232). For example, a hypothetical point The distance D between P and the ground plane Pg (see FIG. 5 ) is greater than the minimum distance, then the position of the point P is taken as the candidate position L c of the obstacle O, and the area corresponding to all the candidate positions L c of the obstacle O in the image I (that is, the upper surface S ou of the obstacle O where the point P is located and the upper part of the obstacle O adjacent to the upper surface S ou ) are taken as the foreground F i of the image I.
  • the distance D can be determined according to the 3D position of the point P The distance from the ground plane P g is obtained.
  • foreground Fi is acquired using a shape-based method based on depth channel data Dd . It should be noted that since the position estimation of the obstacle O based on the depth channel data D d is not robust due to the influence of the sensor noise of the RGB-D camera C, the foreground F i is only used as the initial foreground, which will be used in the next step. Redetermined in the next step (see block 243 of Figure 7).
  • the processing unit 110 further models the distribution of the foreground F i of the image I based on the RGB channel data Dr to obtain the 2D (two-dimensional) position L 2 of the obstacle O (block 240 of FIG. 2A ).
  • the distribution modeling adopts Gaussian distribution modeling, the position of the foreground Fi (pixels in) as input, and uses RGB channel data Dr to model the area outside the foreground Fi as the background Bi ( See Figure 5), thereby forming multiple Gaussian distributions.
  • Mixture Gaussian distribution modeling is also used for foreground F i .
  • the number of Gaussian models eg 5 Gaussian distributions
  • the distribution modeling of the foreground Fi of one image I can be performed to obtain a model for distinguishing obstacles O, and the obtained model can be used to distinguish obstacles O in another image I.
  • FIG. 7 is a schematic block diagram of an example of distribution modeling in the example of ground obstacle detection of FIG. 2A .
  • the processing unit 110 in distribution modeling (block 240 of FIG. 2A ), the processing unit 110 generates a mask M (ie, an initial mask) based on the foreground F i of the obstacle O (block 241 ).
  • the processing unit 110 further uses the mask M to obtain the background Bi (ie, the initial background) of the image I (block 242).
  • the processing unit 110 further models the distribution of the foreground F i and the background B i of the image I based on the RGB channel data Dr to obtain a distribution modeled foreground F m (block 243 ).
  • Each pixel of the image I can be matched to a Gaussian mixture model (GMM) of the foreground F i and the background B i , and the pixel is classified as being closer to the foreground F i or closer to the background B i , thereby obtaining an obstacle.
  • GMM Gaussian mixture model
  • the exact mask M of O is taken as the foreground F i .
  • the background B m after distribution modeling is also obtained.
  • the processing unit 110 further obtains the 2D position L2 of the obstacle O based on the foreground Fm (block 244).
  • the foreground Fi is distributedly modeled using a texture (vision) based approach based on the RGB channel data Dr to obtain a foreground Fm , which is compared to the shape-based approach described above in The location estimation of the obstacle O has improved robustness.
  • the shape-based approach ie, foreground acquisition of block 230 of FIG. 2A
  • the texture-based approach ie, distribution modeling of block 240 of FIG. 2A
  • the processing unit 110 further converts the 2D position L 2 of the obstacle O to the 3D position L 3 of the obstacle O based on the depth channel data D d (block 250 of FIG. 2A ).
  • the 3D position L 3 of the obstacle O may be used to avoid the obstacle O, eg, to re-plan the planned path of the detour to prevent the movement of the mobile machine 100 from being affected by the obstacle O.
  • the 3D position L 3 of the obstacle O obtained by the ground obstacle detection method can be used by the ground obstacle detection sub-module 1211 to obtain information through, for example, indicators (such as arrows, symbols, boxes and masks) displayed on the image I Obstacle O is marked, or provided to other modules/sub-modules (eg, collision avoidance sub-module 1212) for collision avoidance.
  • indicators such as arrows, symbols, boxes and masks
  • the ground obstacle detection method also provides a point cloud of the foreground Fm (for avoiding the obstacle O accordingly), a yes / no signal to indicate whether the obstacle O is present (if the foreground F is acquired) m is yes, this is to avoid the current obstacle O), and/or the plane formula of the ground plane Pg (ie distance and angle relative to the RGB-D camera C).
  • an RGB-D camera is used to detect obstacles on the floor. Unlike existing obstacle avoidance techniques that use other obstacle detection sensors such as lidar, sonar, and infrared sensors that cannot distinguish between obstacles and the ground, this method uses RGB-D cameras to cover the entire ground that the mobile machine moves. In addition, it does not require data training and labeling (for obstacles and floors) when applied to unknown environments, and obtains the location of obstacles by combining the shape and texture features of obstacles as double validation, thereby improving obstacles. Robustness of location estimation. Furthermore, computation (and detection) speed can be optimized by performing shape and texture feature-based verification in parallel.
  • the algorithm of this ground obstacle detection algorithm can detect any obstacle with a height of more than 1 cm within a distance of 1.5 meters (a thin object with a height of less than 1 cm such as carpets, curves, etc. is regarded as non-obstruction). Due to its ability to detect micro/small obstacles, this ground obstacle detection method can provide high safety features for mobile machines.
  • This ground obstacle detection method can be implemented on mobile machines that need to avoid (mini/mini) obstacles, or any computer vision application that needs to detect (mini/mini) objects on the ground surface (eg ground hazard monitoring applications) .
  • Non-transitory memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, solid state drives (SSD), and the like.
  • Volatile memory may include random access memory (RAM), external cache memory, and the like.
  • the processing unit 110 may comprise a central processing unit (CPU), or other general purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or are other programmable logic devices, discrete gates, transistor logic devices, and discrete hardware components.
  • a general-purpose processor can be a microprocessor, and can be any conventional processor.
  • the storage unit 120 (and the above-mentioned memory) may include an internal storage unit such as a hard disk and an internal memory.
  • the storage unit 120 may also include external storage devices such as plug-in hard disks, smart media cards (SMCs), secure digital (SD) cards, and flash memory cards.
  • the exemplary units/modules and methods/steps described in the embodiments may be implemented by software, hardware or a combination of software and hardware. Whether these functions are implemented by software or hardware depends on the specific application and design constraints of the technical solution.
  • the above-mentioned carpet detection method and mobile device 100 can be implemented in other ways.
  • the division of units/modules is only a logical functional division. In actual implementation, other division methods can also be used, that is, multiple units/modules can be combined or integrated into another system, or some features can be ignored or Do not execute.
  • the above-mentioned mutual coupling/connection may be direct coupling/connection or communication connection, or indirect coupling/connection or communication connection through some interfaces/devices, and may also be electrical, mechanical or other forms.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Electromagnetism (AREA)
  • Automation & Control Theory (AREA)
  • Remote Sensing (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Databases & Information Systems (AREA)
  • Computing Systems (AREA)
  • Traffic Control Systems (AREA)

Abstract

一种地面障碍物检测方法以及使用该方法的移动机器，应用于障碍物检测技术领域。通过以下步骤来检测地板上的障碍物：通过RGB-D相机来接收包括深度通道数据和RGB通道数据的图像；基于该深度通道数据来估计对应于该地板的地平面；基于该深度通道数据来获取对应于该地平面的该图像的前景；基于该RGB通道数据来对该图像的该前景进行分布建模，以获取该障碍物的2D位置；以及基于该深度通道数据将该障碍物的2D位置转换为该障碍物的3D位置。

Description

地面障碍物检测方法以及使用该方法的移动机器 技术领域
本发明涉及障碍物检测技术，尤其涉及一种地面障碍物检测方法以及使用该方法的移动机器。
背景技术
随着人工智能(AI)技术的成熟，诸如移动机器人和车辆等的移动机器正在得到广泛使用。由于移动机器通常具有导航能力来执行移动和运输等任务，因此需要实现避障，以防止受到障碍物的影响、从而保证任务的执行。
在现有的避障技术中，使用诸如激光雷达、声纳和红外(IR)传感器等的传统导航传感器来检测障碍物。然而，传感器通常受限于盲点问题，即传感器无法将小型(或微型)障碍物与地面的出现区分开来，而会报告所有物体包括障碍物和地面的出现。因此，它们被安装在移动机器的前方，只感测障碍物、而不感测地面。也就是说，这些传感器将无法覆盖低于其安装高度的小型(或微型)地面障碍物。
相反地，RGB-D传感器(或相机)经济实惠、且可以安装在移动机器上而用来检测前方的地面，并且可以保证其视野能够覆盖移动机器所前往的每个地方。因此，如果提供一种能够区分障碍物与背景的算法，就有可能使用RGB-D传感器来实现更好的微型和小型地面障碍物的检测。
发明内容
本申请提供一种地面障碍物检测方法以及使用该方法的移动机器，用以检测地板上的障碍物，解决前述的现有技术中的地面障碍物检测方法所存在的问 题。
本申请的实施例提供了一种障碍物检测方法，用于使用RGB-D相机来检测地板上的障碍物，该方法包括：
通过RGB-D相机接收图像，其中该图像包括深度通道数据和RGB通道数据；
基于该深度通道数据来估计对应于该地板的地平面；
基于该深度通道数据来获取对应于该地平面的该图像的前景；
基于该RGB通道数据来对该图像的该前景进行分布建模，以获取该障碍物的2D位置；以及
基于该深度通道数据将该障碍物的2D位置转换为该障碍物的3D位置。
本申请的实施例还提供了一种移动机器，包括：
RGB-D相机；
一或多个处理器；以及
一或多个存储器，存储有一或多个计算机程序，该一或多个计算机程序由该一或多个处理器执行，其中该一或多个计算机程序包括多个指令用于：
通过RGB-D相机接收图像，其中该图像包括深度通道数据和RGB通道数据；
基于该深度通道数据来估计对应于该地板的地平面；
基于该深度通道数据来获取对应于该地平面的该图像的前景；
基于该RGB通道数据来对该图像的该前景进行分布建模，以获取该障碍物的2D位置；以及
基于该深度通道数据将该障碍物的2D位置转换为该障碍物的3D位置。
从上述本申请的实施例可知，本申请提供的地面障碍物检测方法使用RGB-D相机取得图像，并对图像的前景进行分布建模、以获取障碍物的位置，从而解决现有技术中难以检测地面上的微型和小型障碍物等问题。
附图说明
为了更清楚地说明本申请的实施例中的技术方案，下面对实施例中或现有技术的描述中所使用的附图进行简要的介绍。在以下的附图中，相同的附图标记在整个图中表示相应的部分。应当理解的是，以下描述中的附图仅为本申请的例子。对于本领域的技术人员来说，在没有创造性劳动的情况下，可以基于这些附图来获得其他的附图。
图1A是本申请的一些实施例中的移动机器的导航的示意图。
图1B是说明图1A的移动机器的示意框图。
图2A是图1B的移动机器的地面障碍物检测的例子的示意框图。
图2B是图2A的地面障碍物检测的示意图。
图3是图1B的移动机器的RGB-D相机所拍摄的图像的示意图。
图4是图2A的地面障碍物检测的例子中的地平面估计的例子的示意框图。
图5是图2A的地面障碍物检测的例子中的地平面估计和前景获取的例子的示意图。
图6是图2A的地面障碍物检测的例子中的前景获取的例子的示意框图。
图7是图2A的地面障碍物检测的例子中的分布建模的例子的示意框图。
具体实施方式
为使本申请的目的、特征和优点更加明显易懂，下面将结合附图对本申请的实施例中的技术方案进行清楚、完整的描述。显然，所描述的实施例仅仅是本申请的一部分实施例，而非全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有作出创造性劳动的前提下所获得的所有其他实施例，都在本申请所保护的范围内。
应当理解的是，当在本申请和所附的权利要求中使用时，术语“包括”、“包含”、“具有”及其变体表示所述特征、整体、步骤、操作、元素和/或组 件的存在，但不排除可以存在或添加一或多个其他特征、整体、步骤、操作、元素、组件和/或其集合。
还应当理解的是，在本申请的说明书中所使用的术语仅仅是出于描述特定实施例的目的，而非限制本申请的范围。如同在本申请的说明书和所附的权利要求中所使用的那样，除非上下文清楚地指明了其他情况，否则单数形式的“一”、“一个”以及“该”意在包括复数形式。
还应当进一步理解，在本申请说明书和所附的权利要求中所使用的术语“和/或”是指相关列出的项中的一或多个的任何组合以及所有可能组合，并且包括这些组合。
本申请中的术语“第一”、“第二”、“第三”仅仅是出于描述的目的，而不能理解为指示或暗示了相对重要性、或是暗示了所指的技术特征的数量。由此，由“第一”、“第二”、“第三”所限定的特征可以显式或隐式地包括其中的至少一个技术特征。在本申请的描述中，“多个”的含义是至少两个，例如两个、三个等，除非另有明确的定义。
在本申请的说明书中叙述的“一个实施例”或“一些实施例”等意味着在本申请的一或多个实施例中可以包括与该实施例的描述内容相关的特定特征、结构或特点。由此，在说明书的不同地方出现的语句“在一个实施例中”、“在一些实施例中”、“在其他一些实施例中”、“在另外一些实施例中”等不是意味着所描述的实施例应该被所有其他实施例所引用，而是被“一或多个但不是所有其他实施例所引用”，除非另有特别强调。
本申请涉及地面障碍物的检测，可以被移动机器使用。如本文所用，术语“移动机器”是指具有在其所处环境中四处移动的能力的机器，例如移动机器人或车辆。术语“障碍物”是指在地板上而可能影响移动机器的移动的物体。术语“传感器”是指诸如图像传感器和陀螺仪等的设备、模块、机器或子系统，其目的是检测其环境中的事件或变化、并将信息发送到其他电子设备(例如处 理器)。术语“导航”是指监视和控制移动机器从一个地方到另一个地方的移动的过程。术语“防撞”是指在导航过程中防止或减轻碰撞的严重程度。
图1A是本申请的一些实施例中的移动机器100的导航的示意图。移动机器100(例如移动机器人和车辆)在其环境(例如办公室)中被导航、以在地板F上移动，同时检测诸如坠落物体、垃圾、家具、宠物或类似物(例如墙壁W)等的在地板F上的地面障碍物O，以防止其移动受到诸如阻碍、减速、绊倒或滑倒等影响，或避免诸如碰撞和跌倒等危险情况。在室内导航中，移动设备100沿着起点(例如移动机器100所在的位置)和目的地(例如用户或移动设备100的导航/操作系统所指定的导航目标的位置)之间的路径从起点导航到目的地，同时需要避开地板F上的诸如坠落的物体、垃圾、以至于墙壁等障碍物O。在一些实施例中，为了实现移动设备100的导航，需要构建环境地图、确定移动机器100在环境中的当前位置，而且需要规划移动机器100从起点移动到目的地的路径。、
在一些实施例中，移动机器100的导航和/或地面障碍物检测可以由移动机器100本身(例如移动机器100上的控制接口)或控制装置200(例如移动机器100的遥控器)通过例如提供对移动机器100的导航和/或地面障碍物检测的请求来启动。控制装置200可以是例如智能手机、平板电脑、笔记本电脑、台式电脑或其他电子设备。
图1B是说明图1A的移动机器100的示意框图。移动机器100可以是移动机器人，例如轮式机器人或人形机器人，其可以包括通过一或多条通信总线或信号线L进行通信的处理单元110、存储单元120和控制单元130。需要说明的是，移动机器100只是移动设备的一个例子，移动机器100可以具有比上面或下面所示更多或更少的组件(例如单元、子单元和模块)，可以组合两个或更多个组件，或者可以具有不同的组件配置或排列。处理单元110执行存储在存储单元120中的各种(各组)指令，这些指令可以是软件程序的形式，以执 行移动机器100的各种功能并处理相关数据，其可以包括一或多个处理器(例如中央处理器(CPU))。存储单元120可以包括一或多个存储器(例如高速随机存取存储器(RAM)和非暂时性存储器)、一或多个存储器控制器，以及一或多个非暂时性计算机可读存储介质(例如固态状态驱动器(SSD)或硬盘)。控制单元130可以包括各种控制器(例如相机控制器、显示器控制器和物理按钮控制器)和用于耦合移动机器100的输入/输出外围设备到处理单元110和存储单元120的外围设备接口，例如处理单元110的外部端口(例如USB)、无线通信电路(例如RF通信电路)、音频电路(例如扬声器电路)、传感器(例如陀螺仪和相机)。在一些实施例中，存储单元120可以包括导航模块121，用于实现与移动机器100的导航(和地面障碍物检测)相关的功能(例如地图构建、路径规划和地面障碍物检测)，其可以存储在一个或多个存储器(以及一个或多个非暂时性计算机可读存储介质)中。在其他实施例中，移动机器100可以是交通工具，例如汽车。
移动设备100的存储单元120中的导航模块121可以是(移动机器100的操作系统的)软件模块，其具有用于实现移动机器100的导航的指令I n(例如用来驱动移动机器100的电机M以移动移动机器100的指令)、地面障碍物检测子模块1211和防撞子模块1212。地面障碍物检测子模块1211可以是具有于对移动机器100进行地面障碍物检测的指令I o的软件模块，从而检测地面上的障碍物。防撞子模块1212可以是具有用于实现移动机器100的防撞的指令I c的软件模块，从而避免与障碍物发生碰撞。地面障碍物检测子模块1211和防撞子模块1212可以是独立于导航模块121的指令I n或其他子模块的子模块，或者是用于实现移动机器100的导航的指令I n的一部分。地面障碍物检测子模块1211还可以具有与移动机器100的地面障碍物检测相关的数据(例如输入/输出数据和临时数据)，其可以存储在一或多个存储器中、并由处理单元110访问。在一些实施例中，地面障碍物检测子模块1211可以是存储单元120中与导航模 块121分离的模块，也可以是集成在防撞子模块1212中的模块。
在一些实施例中，指令I n可以包括用于实现移动机器100的地面障碍物检测和碰撞避免的指令。在其他实施例中，导航模块121可以是通过一或多条通信总线或信号线L与处理单元110、存储单元120和控制单元130进行通信的导航单元，还可包括用于存储指令I n、地面障碍物检测子模块1211和防撞子模块1212的一或多个存储器(例如高速随机存取存储器(RAM)和非暂态存储器)以及一个或多个处理器(例如微處理器(MPU)和微控制器(MCU))用于执行存储的指令I n、I o和I c、以实现移动机器100的导航。
移动机器100还可以包括传感器子单元133。传感器子单元133可以包括一组传感器和相关控制器，例如RGB-D相机C和陀螺仪G(或者惯性测量单元(inertial measurement unit，IMU))，用于检测其所处的环境、以实现其导航。RGB-D相机C可以安装在移动设备100的前面以面向移动机器100移动的方向，使得其视野V(见图2B)能够覆盖移动机器100所移动的每个地方。传感器子单元133通过一或多条通信总线或信号线L与控制单元130进行通信，该一或多条通信总线或信号线可以与上述的一或多条通信总线或信号线L相同、或至少部分不同。在其他实施例中，在导航模块121为上述导航单元的情况下，传感器子单元133可以通过一或多条通信总线或信号线与导航单元进行通信，该通信总线或信号线可以与上述的一或多条通信总线或信号线L相同、或至少部分不同。此外，传感器子单元133可以只是抽象组件，用于表示移动机器100的组件之间的逻辑关系。
移动机器100还可以包括通信子单元131和致动子单元132。通信子单元131和致动子单元132通过一或多条相同的通信总线或信号线与控制单元130进行通信，该一或多条通信总线或信号线可以与上述的一或多条通信总线或信号线L相同、或至少部分不同。通讯子单元131耦接到移动机器100的通讯接口，诸如供移动机器100通过网络与控制装置200进行通信的网络接口1311、 I/O接口1312(例如物理按钮)等。致动子单元132耦合到用于实现移动机器100的移动的组件/设备，以驱动移动机器100的轮子和/或关节的电机M。通信子单元131可以包括用于移动机器100的上述通信接口的控制器，致动子单元132可以包括用于实现移动机器100的移动的上述组件/设备的控制器。在其他实施例中，通信子单元131和/或致动子单元132可以只是抽象组件，用以表示移动机器100的组件之间的逻辑关系。
在一些实施例中，地面障碍物检测子模块1211、防碰撞子模块1212、传感器子单元133和电机M(以及与电机M耦合的移动机器100的轮子和/或关节)共同组成一个(导航)系统，实现地图构建、(全局和局部)路径规划和电机驱动，从而实现移动机器100的导航。
在一些实施例中，图1B中所示的各种组件可以以硬件、软件或硬件和软件的组合来实现。处理单元110、存储单元120、控制单元130、导航模块121和其他单元/子单元/模块中的两个或更多个可以实现在单个芯片或电路上。在其他实施例中，它们中的至少一部分可以在单独的芯片或电路上实现。
图2A是图1B的移动机器100的地面障碍物检测的例子的示意框图。在一些实施例中，以例如将对应于移动机器100的地面障碍物检测方法的指令(组)I o存储为存储单元120中的导航模块121、并通过处理单元110执行所存储的指令I o的方式在移动机器100中实现该地面障碍物检测方法。该地面障碍物检测可以响应于来自例如移动机器100本身或者控制装置200(的导航/操作系统)的移动机器100的地面上障碍物检测的请求而执行，然后也可以例如每隔特定的时间间隔(例如1秒)和/或在重新规划计划路径时重新执行，直到移动机器100的导航结束。图2B是图2A的地面障碍物检测示意图。如图2B所示，RGB-D相机C的视野V覆盖了地面障碍物O和地面F。通过地面障碍物检测，将检测到障碍物O、并获取障碍物O的3D(三维)位置L 3。
根据该地面障碍物检测方法，处理单元110通过RGB-D相机C接收图像I (图2A的框210)。图3是图1B的移动机器100的RGB-D相机C所拍摄的图像I的示意图。图像I是移动机器100所导航的图1A的环境中的场景的图像。图像I包括深度通道数据D d和RGB通道数据D r。深度通道数据D d包括图像I的深度通道，用于表示到图像I中的场景对象(例如地面上的障碍物O、地面F和墙壁W)的距离。RGB通道数据D r包括图像I的红色通道、绿色通道和蓝色通道，用于表示构成图像I的颜色。
处理单元110进一步基于深度通道数据D d来估计对应于地板F的地平面P g(图2A的框220)。地平面P g是图像I中的估计平面，表示障碍物O所在的地板F。在一些实施例中，在接收到多个图像I的情况下，可以基于图像I的深度通道数据D d来估计地平面P g，例如通过估计每个图像I的地平面P g、并取所估计的其中一个地平面P g(例如出现在多个图像I中的地平面P g)。图4是图2A的地面障碍物检测的例子中的地平面估计的例子的示意框图。在一些实施例中，在地平面估计(图2A的框220)中，由于RGB-D相机C的视野的全域中所含的3D空间中的大量点可能包括前景和背景，这些点不属于单一平面。因此这些点被划分为多个小的块(patch)(图4的框221)，其中每个块代表场景中总是可以近似地表示深度通道数据D d中的平面P s的一个小区域。作为例子，基于图像I的坐标系中的x和y坐标和深度通道数据D d中的距离信息(即图像I的坐标系中的z坐标)，深度通道数据D d中的每个平面状点云(即像素集)簇被识别为一个平面P s。处理单元110进一步基于深度通道数据D d获取每个平面P s相对于RGB-D相机C的夹角θ p(即图5中的“平面-相机角度θ p”)(图4的框222)。然后，可以基于相邻的小块平面P s的角度的相似性和高度的差(假设属于地板F的所有小块平面P s之间的角度相似、且相对于RGB-D相机C的高度相似)，通过将相邻的小块平面P s分组为一个大平面来计算地平面P g。例如当地板有较小的曲率或不完全光滑时，只要相邻的小块具有相似的夹角，就会认为它们属于一个较大的平面。每个小平面P s的角度可以从平面 拟合过程中计算出来。平面几何模型由法线向量n＝[a，b，c] T和距离d描述，使得对平面P s上的点p＝[x，y，z] T来说n·p+d＝0。给定一组点作为输入，可以解析出满足大多数点的法线向量n。
处理单元110进一步在深度通道数据D d中寻找地平面P g(参见图5)(图4的框223)。图5是图2A的地面障碍物检测的例子中的地平面估计和前景获取的例子的示意图，其中假设地平面P g与地板F相同。P g被估计为摄像机C的视野中的所谓的主平面，即场景中最大的连接平面。例如当RGB-D相机C相对于地板F具有更大的高度时，为了更好地感测地板F、而非向前(即与地板F平行)拍摄，RGB-D相机C可以具有向下倾斜角θ c而以相对于地板F的倾斜角进行拍摄，因此视野中的主要平面结构是地面/地板，而不是墙壁或其他结构。
在一些实施例中，具有相似夹角的相邻平面P s(例如具有不大于预定差值(例如5度)的夹角的相邻平面P s)连接为一个单一平面，具有被作为属于地平面P g的所有连接小平面P s的平均值的夹角的最大连接单一平面被作为地平面P g。作为例子，与障碍物O的下表面S ol平行的平面P s连接为较大的单一平面，而位于障碍物O的上表面S ou的平面P s连接为较小的单一平面或的多个平面(当上表面S ou曲率大时)，且上表面S ou处的一个或多个平面和下表面S ol处的平面P s由于它们的夹角θ p或相对高度的大差异而被断开。通过与障碍物O的下表面S ol平行的平面P s连接的较大的单一平面将被作为接地平面P g。
处理单元110进一步基于深度通道数据D d获取对应于地平面P g的图像I的前景F i(即图5中的初始前景)(图2A的框230)。将获得前景F i(中的像素)的位置。在一些实施例中，在接收到多个图像I的情况下(参见图2A的框210)，可以基于图像I的深度通道数据D d获得前景F i，例如通过获得每个图像I的地平面P g并取其中一个(最大的)所获得的前景F i。
图6是图2A的地面障碍物检测的例子中的前景获取的例子的示意框图。在一些实施例中，在前景获取(框230)中，处理单元110获取深度通道数据 D d中的每个点P(参见图5)的位置，该位置与地平面P g相距至少一最小距离(即将距离地平面P g的距离大于或等于最小距离(例如1厘米)的每个点P作为障碍物O的候选位置L c(框231)。候选位置L c可以包括每个点P的坐标(例如笛卡尔坐标(x，y，z))。处理单元110进一步将图像I中对应于障碍物O的所有候选位置L c的区域作为图像I的前景F i(框232)。例如假设点P与地平面P g的距离D(见图5)大于最小距离，则点P的位置作为障碍物O的候选位置L c，且图像I中对应于障碍物O的所有候选位置L c的区域(即点P所在的障碍物O的上表面S ou和与上表面S ou相邻的障碍物O的上部)被作为取图像I的前景F i。其中，距离D可以根据点P的3D位置与地平面P g之间的距离来获得。
在前景获取(框230)中，使用基于深度通道数据D d的基于形状的方法来获取前景F i。需要注意的是，由于基于深度通道数据D d的障碍物O位置估计因为受到RGB-D相机C的传感器噪声的影响而不具有鲁棒性，因此前景F i仅作为初始前景，其将在接下来的步骤中重新确定(参见图7的框243)。
处理单元110进一步基于RGB通道数据D r对图像I的前景F i进行分布建模、以获取障碍物O的2D(二维)位置L 2(图2A的框240)。在一些实施例中，分布建模采用高斯分布建模，前景F i(中的像素)的位置作为输入，并使用RGB通道数据D r将前景F i之外的区域建模为背景B i(见图5)，从而形成多个高斯分布。混合高斯分布建模也用于前景F i。可以调整高斯模型的数量(例如5个高斯分布)以实现鲁棒性和速度之间的平衡。另外，在接收到多幅图像I的情况下(见框220)，因为分布建模与地面障碍物检测方法的其他步骤(见图2A的框210-框230和框250)相比比较耗时，可以对其中一幅图像I的前景F i进行分布建模、得到用于区分障碍物O的模型，所得到的模型可以用来区分另一幅图像I中的障碍物O。
图7是图2A的地面障碍物检测的例子中的分布建模的例子的示意框图。在一些实施例中，在分布建模(图2A的框240)中，处理单元110基于障碍物 O的前景F i生成遮罩M(即初始遮罩)(框241)。处理单元110进一步使用遮罩M来获取图像I的背景Bi(即初始背景)(框242)。处理单元110进一步基于RGB通道数据D r对图像I的前景F i和背景B i进行分布建模，以获取分布建模后的前景F m(框243)。图像I的每个像素可以匹配到前景F i和背景B i的高斯混合模型(Gaussian mixture model，GMM)，并将该像素分类为更接近前景F i或更接近背景B i，从而得到障碍物O的精确遮罩M作为前景F i。此外，还获取了分布建模后的背景B m。处理单元110进一步基于前景F m获取障碍物O的2D位置L 2(框244)。
在分布建模(框240)中，使用基于RGB通道数据D r的基于纹理(视觉)的方法对前景F i进行分布建模、以获取前景F m，其与上述基于形状的方法相比在障碍物O的位置估计具有改进的鲁棒性。基于形状的方法(即图2A的框230的前景获取)和基于纹理的方法(即图2A的块240的分布建模)可以并行执行、以优化计算速度。
处理单元110进一步基于深度通道数据D d将障碍物O的2D位置L 2转换成障碍物O的3D位置L 3(图2A的框250)。在一些实施例中，障碍物O的3D位置L 3可用于避开障碍物O，例如重新规划绕道的计划路径、以防止移动机器100的移动受到障碍物O的影响。
通过地面障碍物检测方法获得的障碍物O的3D位置L 3，可以被地面障碍物检测子模块1211使用而通过例如显示在图像I上的指示符(例如箭头、符号、框和遮罩)来标记障碍物O，或者提供给其他模块/子模块(例如防撞子模块1212)以实现防撞。在一些实施例中，地面障碍物检测方法还提供前景F m的点云(用于相应地避开障碍物O)、用来表示障碍物O是否出现的是/否信号(如果获取了前景F m则为是，这是为了避开当前的障碍物O)、和/或地平面P g的平面公式(即相对于RGB-D相机C的距离和角度)。
在地板上障碍物检测方法中，使用一个RGB-D相机来检测地板上障碍物。 与现有的使用激光雷达、声纳和红外传感器等无法区分障碍物和地面的其他障碍物检测传感器的避障技术不同，该方法使用RGB-D相机覆盖了移动机器所移动的整个地面。此外，它在应用于未知环境时不需要数据的训练和标记(针对障碍物和地板)等工作，并且通过结合障碍物的形状和纹理特征作为双重验证来获取障碍物的位置，借以提高障碍物位置估计的鲁棒性。此外，可以通过并行执行基于形状和纹理特征的验证来优化计算(和检测)速度。根据使用D435i类型的RealSense深度相机的实验，该地面障碍物检测算法的算法可以检测到1.5米距离内的任何1厘米以上高度的障碍物(地毯、曲线等低于1厘米高度的薄物体视为非障碍物)。由于其检测微型/小型的障碍物的能力，该地面障碍物检测方法能够为移动机器提供高安全的特性。
该地面障碍物检测方法可以在需要避开(微型/小型)障碍物的移动机器、或需要检测地表上的(微型/小型)的物体任何计算机视觉应用程序(例如地面危险监测应用程序)上实现。
本领域技术人员可以理解，上述实施例中的方法的全部或部分可以通过使用一或多个计算机程序来指示相关硬件而实现。此外，一或多个程序可以存储在非暂时性计算机可读存储介质中。当执行一或多个程序时，执行上述实施例中对应的方法的全部或部分。对存储、存储器、数据库或其他介质的任何引用可以包括非暂时性和/或暂时性存储器。非暂时性存储器可以包括只读存储器(ROM)、可编程ROM(PROM)、电可编程ROM(EPROM)、电可擦除可编程ROM(EEPROM)、闪存、固态驱动器(SSD)等。易失性存储器可以包括随机存取存储器(RAM)、外部高速缓存存储器等。
处理单元110(以及上述处理器)可以包括中央处理单元(CPU)，或者是其他通用处理器、数字信号处理器(DSP)、专用集成电路(ASIC)、现场可编程门阵列(FPGA)，或者是其他可编程逻辑器件、分立门、晶体管逻辑器件和分立硬件组件。通用处理器可以是微处理器，可以是任何常规处理器。存储单元120(以 及上述存储器)可以包括诸如硬盘和内部存储器的内部存储单元。存储单元120还可以包括外部存储设备，例如插入式硬盘、智能媒体卡(SMC)、安全数字(SD)卡和闪存卡。
实施例中描述的示例性单元/模块和方法/步骤可以通过软件、硬件或者软件和硬件的结合来实现。这些功能究竟是通过软件实现还是通过硬件实现，取决于技术方案的具体应用和设计约束。上述的地毯检测方法及移动机100可以通过其他方式实现。例如单元/模块的划分仅仅是逻辑上的功能划分，在实际实现中还可以采用其他划分方式，即可以将多个单元/模块组合或集成到另一个系统中，或者某些特征可以被忽略或不执行。此外，上述相互耦合/连接可以是直接耦合/连接或通信连接，也可以是通过一些接口/设备的间接耦合/连接或通信连接，还可以是电气、机械或其他形式。
上述实施例仅用于说明本发明的技术方案，并不用于限制本发明的技术方案。虽然本发明已经结合上述实施例进行了详细说明，但是上述各个实施例中的技术方案仍然可以进行修改，或者部分技术特征可以等效替换，从而使这些修改或替换没有使相应技术方案的实质脱离本发明各实施例的技术方案的精神和范围，均应包含在本发明保护的范围之内。

Claims (10)

  1. 一种障碍物检测方法，用于使用RGB-D相机来检测地板上的障碍物，该方法包括：
    通过RGB-D相机接收图像，其中该图像包括深度通道数据和RGB通道数据；
    基于该深度通道数据来估计对应于该地板的地平面；
    基于该深度通道数据来获取对应于该地平面的该图像的前景；
    基于该RGB通道数据来对该图像的该前景进行分布建模，以获取该障碍物的2D位置；以及
    基于该深度通道数据将该障碍物的2D位置转换为该障碍物的3D位置。
  2. 如权利要求1所述的方法，其中所述基于该深度通道数据来估计对应于该地板的地平面包括：
    将该深度通道数据中的多个点划分为一或多个块，其中每个块代表该深度通道数据中的一个平面；
    基于该深度通道数据获取每个该一或多个平面相对于该RGB-D相机的夹角；以及
    基于该一或多个平面的该夹角形成对应于该一或多个平面中的至少一个的该地平面。
  3. 根据权利要求2所述的方法，其中所述基于该深度通道数据获取每个该一或多个平面相对于该RGB-D相机的夹角包括：
    通过该平面的法线向量获取每个该一或多个平面与该RGB-D相机之间的该夹角，其中该法线向量是通过拟合该深度通道数据中的像素而获取。
  4. 如权利要求2所述的方法，其中所述基于该一或多个平面的该夹角形成对应于该一或多个平面中的至少一个的该地平面包括：
    将相邻的该些平面连接为单一平面，其中相邻的该平面的该夹角之间的差 不大于预定差；以及
    将最大的该单一平面作为该地平面。
  5. 如权利要求1所述的方法，其中所述基于该深度通道数据来获取对应于该地平面的该图像的该前景包括：
    将该深度通道数据中与该地平面相距至少有一最小距离的各个点的位置作为该障碍物的候选位置；以及
    将该图像中与该障碍物的所有该候选位置相对应的区域作为该图像的该前景。
  6. 如权利要求1所述的方法，其中所述基于该RGB通道数据来对该图像的该前景进行该分布建模，以获取该障碍物的2D位置，包括：
    根据该障碍物的该前景生成遮罩；
    使用该遮罩获取该图像的背景；
    基于该RGB通道数据来对该图像的该前景和该背景进行该分布建模；以及
    基于该图像的该前景获取该障碍物的该2D位置。
  7. 根据权利要求1所述的方法，其中该3D位置包括对应于该障碍物的多个坐标。
  8. 一种移动机器，包括：
    RGB-D相机；
    一或多个处理器；以及
    一或多个存储器，存储有一或多个计算机程序，该一或多个计算机程序由该一或多个处理器执行，其中该一或多个计算机程序包括多个指令用于：
    通过RGB-D相机接收图像，其中该图像包括深度通道数据和RGB通道数据；
    基于该深度通道数据来估计对应于该地板的地平面；
    基于该深度通道数据来获取对应于该地平面的该图像的前景；
    基于该RGB通道数据来对该图像的该前景进行分布建模，以获取该障碍物的2D位置；以及
    基于该深度通道数据将该障碍物的2D位置转换为该障碍物的3D位置。
  9. 根据权利要求8所述的移动机器，其中所述基于该深度通道数据来估计对应于该地板的地平面包括：
    将该深度通道数据中的多个点划分为一或多个块，其中每个块代表该深度通道数据中的一个平面；
    基于该深度通道数据获取每个该一或多个平面相对于该RGB-D相机的夹角；以及
    基于该一或多个平面的该夹角形成对应于该一或多个平面中的至少一个的该地平面。
  10. 一种计算机可读存储介质，存储有一或多个计算机程序，其中该一或多个计算机程序包括多个指令，当该多个指令由具有多个传感器的移动机器执行时，使该移动机器：
    通过RGB-D相机接收图像，其中图像包括深度通道数据和RGB通道数据；
    基于该深度通道数据估计对应于该地板的地平面，
    基于该深度通道数据来获取对应于该地平面的该图像的前景；
    基于该RGB通道数据来对该图像的该前景进行分布建模，以获取该障碍物的2D位置；以及
    基于该深度通道数据将该障碍物的2D位置转换为该障碍物的3D位置。
PCT/CN2022/082148 2021-04-26 2022-03-22 地面障碍物检测方法以及使用该方法的移动机器 WO2022227939A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202280003657.6A CN115516523A (zh) 2021-04-26 2022-03-22 地面障碍物检测方法以及使用该方法的移动机器

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US17/239,697 2021-04-26
US17/239,697 US11734850B2 (en) 2021-04-26 2021-04-26 On-floor obstacle detection method and mobile machine using the same

Publications (1)

Publication Number Publication Date
WO2022227939A1 true WO2022227939A1 (zh) 2022-11-03

Family

ID=83694412

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/082148 WO2022227939A1 (zh) 2021-04-26 2022-03-22 地面障碍物检测方法以及使用该方法的移动机器

Country Status (3)

Country Link
US (1) US11734850B2 (zh)
CN (1) CN115516523A (zh)
WO (1) WO2022227939A1 (zh)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2015049776A (ja) * 2013-09-03 2015-03-16 国立大学法人 東京大学 画像処理装置、画像処理方法及び画像処理プログラム
CN107220632A (zh) * 2017-06-12 2017-09-29 山东大学 一种基于法向特征的路面图像分割方法
CN108245385A (zh) * 2018-01-16 2018-07-06 曹醒龙 一种帮助视障人士出行的装置
CN110304386A (zh) * 2019-06-28 2019-10-08 北京极智嘉科技有限公司 机器人及机器人丢码后的重新定位方法
CN110399807A (zh) * 2019-07-04 2019-11-01 深圳前海达闼云端智能科技有限公司 检测地面障碍物的方法、装置、可读存储介质及电子设备
CN110956137A (zh) * 2019-12-02 2020-04-03 深圳市镭神智能系统有限公司 点云数据的目标检测方法、系统及介质

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140363073A1 (en) * 2013-06-11 2014-12-11 Microsoft Corporation High-performance plane detection with depth camera data
US20220277469A1 (en) * 2019-09-23 2022-09-01 Intel Corporation Scene retrieval for computer vision

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2015049776A (ja) * 2013-09-03 2015-03-16 国立大学法人 東京大学 画像処理装置、画像処理方法及び画像処理プログラム
CN107220632A (zh) * 2017-06-12 2017-09-29 山东大学 一种基于法向特征的路面图像分割方法
CN108245385A (zh) * 2018-01-16 2018-07-06 曹醒龙 一种帮助视障人士出行的装置
CN110304386A (zh) * 2019-06-28 2019-10-08 北京极智嘉科技有限公司 机器人及机器人丢码后的重新定位方法
CN110399807A (zh) * 2019-07-04 2019-11-01 深圳前海达闼云端智能科技有限公司 检测地面障碍物的方法、装置、可读存储介质及电子设备
CN110956137A (zh) * 2019-12-02 2020-04-03 深圳市镭神智能系统有限公司 点云数据的目标检测方法、系统及介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
YANG SHUAI: "Research on Obstacle Avoidance Strategy of Mobile Robot Based on RGBD Image", MASTER THESIS, TIANJIN POLYTECHNIC UNIVERSITY, CN, no. 2, 15 February 2018 (2018-02-15), CN , XP055982476, ISSN: 1674-0246 *

Also Published As

Publication number Publication date
US11734850B2 (en) 2023-08-22
CN115516523A (zh) 2022-12-23
US20220343530A1 (en) 2022-10-27

Similar Documents

Publication Publication Date Title
TWI662388B (zh) 機器人之避障控制系統及方法
US20200249032A1 (en) Apparatus and method for updating high definition map for autonomous driving
CN110874100A (zh) 用于使用视觉稀疏地图进行自主导航的系统和方法
CN110119698B (zh) 用于确定对象状态的方法、装置、设备和存储介质
WO2022078467A1 (zh) 机器人自动回充方法、装置、机器人和存储介质
CN108345836A (zh) 用于自主车辆的标志识别
WO2021046716A1 (zh) 目标对象的检测方法、系统、设备及存储介质
CN110807350A (zh) 用于面向扫描匹配的视觉slam的系统和方法
CN111814752B (zh) 室内定位实现方法、服务器、智能移动设备、存储介质
WO2023016271A1 (zh) 位姿确定方法、电子设备及可读存储介质
US10165246B2 (en) Method and device for processing stereoscopic data
EP3716210B1 (en) Three-dimensional point group data generation method, position estimation method, three-dimensional point group data generation device, and position estimation device
US11748998B1 (en) Three-dimensional object estimation using two-dimensional annotations
CN112097732A (zh) 一种基于双目相机的三维测距方法、系统、设备及可读存储介质
US10902610B2 (en) Moving object controller, landmark, and moving object control method
US11460855B1 (en) Systems and methods for sensor calibration
KR20120078339A (ko) 이동 로봇의 이미지 기반 동시적 위치 인식 및 지도 작성 방법
WO2022227939A1 (zh) 地面障碍物检测方法以及使用该方法的移动机器
WO2019183928A1 (zh) 一种室内机器人定位的方法及机器人
WO2022134863A1 (zh) 防撞方法、移动机器及存储介质
WO2022227632A1 (zh) 基于图像的轨迹规划方法和运动控制方法以及使用该些方法的移动机器
Kume et al. Vehicle localization along a previously driven route using an image database
Poomarin et al. Automatic docking with obstacle avoidance of a differential wheel mobile robot
Sun et al. Detection and state estimation of moving objects on a moving base for indoor navigation
WO2022160101A1 (zh) 朝向估计方法、装置、可移动平台及可读存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22794415

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE