WO2022017320A1

WO2022017320A1 - Obstacle information obtaining method, obstacle avoidance method, moving apparatus, and computer-readable storage medium

Info

Publication number: WO2022017320A1
Application number: PCT/CN2021/107100
Authority: WO
Inventors: 谢亮; 王果
Original assignee: 影石创新科技股份有限公司
Priority date: 2020-07-21
Filing date: 2021-07-19
Publication date: 2022-01-27
Also published as: CN112016394A

Abstract

An obstacle information obtaining method, a moving apparatus implementing the method, and a computer-readable storage medium. The method comprises: obtaining at least two images of the same scene around the moving apparatus; respectively storing the at least two images as a high-resolution picture and a low-resolution picture; detecting a suspected obstacle according to the low-resolution picture of the at least two images; and obtaining detailed information of the suspected obstacle according to the high-resolution picture of the at least two images. According to the method, the obtained at least two images are respectively stored as the high-resolution picture and the low-resolution picture, then the suspected obstacle is detected according to the low-resolution picture of the at least two images, and the detailed information of the suspected obstacle is obtained according to the high-resolution picture of the at least two images. In this way, the method only performs high-precision matching and distance calculation on a picture having the suspected obstacle, thereby improving the efficiency and speed of obtaining the obstacle information.

Description

Obstacle information acquisition method, obstacle avoidance method, mobile device and computer-readable storage medium

technical field

The embodiments of the present application belong to the field of environment perception of mobile devices, and specifically relate to a method for acquiring obstacle information, a method for avoiding obstacles, a mobile device, and a computer-readable storage medium.

Background technique

Depth imaging is a technology that uses imaging equipment to extract scene depth information and represent the depth information as a depth image. This technology can be combined with target detection, target recognition, image segmentation and other technologies, and is applied in the fields of intelligent video surveillance, driverless cars, intelligent transportation, security and automatic robot control.

In order to obtain depth information of environmental objects, existing technologies mainly include stereo vision, structured light, and TOF. Among them, structured light is not very effective when the light intensity is high, and TOF cannot be widely used due to high product cost.

technical problem

The local algorithms (including local area matching and local feature matching) of the existing stereo matching algorithms for stereo vision have insufficient matching accuracy, while the global algorithms (mainly including the graph cut method) have too long running time and require more processor configuration. high defect. The purpose of the present invention is to provide an obstacle information acquisition method, an obstacle avoidance method, a mobile device and a computer-readable storage medium, so as to improve the efficiency of obstacle information acquisition and the efficiency of the mobile device to avoid obstacles.

technical solutions

In a first aspect, the present invention provides a method for obtaining obstacle information, the method comprising: obtaining at least two images of the same scene around a mobile device; saving the at least two images as a high-resolution picture and a low-resolution picture respectively ; Detect suspected obstacles according to low-resolution pictures of at least two images; obtain detailed information of suspected obstacles according to high-resolution pictures of at least two images.

In addition, the present invention also provides a mobile device and a computer-readable storage medium based on the above method for obtaining obstacle information.

In a second aspect, the present invention provides an obstacle avoidance method for a mobile device, the method comprising: setting a safe operating area centered on the mobile device; acquiring at least two images of the same scene around the mobile device; The images are saved as high-resolution pictures and low-resolution pictures respectively; the suspected obstacles are detected according to the low-resolution pictures of at least two images; the detailed information of the suspected obstacles is obtained according to the high-resolution pictures of at least two images; Obstacle details to perform obstacle avoidance actions.

In addition, the present invention also provides a mobile device and a computer-readable storage medium based on the above obstacle avoidance method.

beneficial effect

The present invention saves the acquired at least two images as high-resolution pictures and low-resolution pictures respectively, then detects suspected obstacles according to the low-resolution pictures of the at least two images, and then detects the suspected obstacles according to the high-resolution pictures of the at least two images. Get detailed information on suspected obstacles. In the above manner, the present invention only performs high-precision matching and distance calculation for pictures with suspected obstacles, which improves the efficiency and speed of obtaining obstacle information.

Description of drawings

FIG. 1 is a flowchart of a method for obtaining obstacle information in a specific embodiment of the present invention.

FIG. 2 is a flowchart of detecting a suspected obstacle in a specific embodiment of the present invention.

FIG. 3 is a schematic diagram of a region of interest in a specific embodiment of the present invention.

FIG. 4 is a schematic block diagram of a mobile device according to an embodiment of the present invention.

FIG. 5 is a flowchart of an obstacle avoidance method for a mobile device in a specific embodiment of the present invention.

FIG. 6 is a schematic diagram of a safe operating area of a mobile device according to an embodiment of the present invention.

Embodiments of the present invention

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

The specific implementation of the present invention is described in detail below in conjunction with specific embodiments:

Example 1:

As shown in FIG. 1 , it is a flowchart of a preferred embodiment of a method for obtaining obstacle information according to the present invention, and the specific steps are as follows.

Step 1: Acquire at least two images of the same scene around the mobile device.

Specifically, the mobile device may be a drone or a mobile cart. For example, binocular cameras are installed on the upper and lower sides, the front and rear ends, and the left and right sides of the drone to obtain two images of the drone in all directions; another example, on the top, front and rear ends of the mobile car Binocular cameras are installed on the five sides of the left and right sides respectively to obtain two images of the mobile car in five directions except the ground direction. In addition, the number of binocular cameras can also be increased or decreased according to the shape of the mobile device, the positions of the two lenses of the binocular camera, or the field of view of the lenses. The two lenses are installed at the upper and lower ends, so that the upper and lower lenses on the same side form a crossed field of view. When the field of view of the lens is large (such as a fisheye lens), the thickness of the mobile device can be appropriately increased. At this time, the upper and lower fisheye lenses located on the same side can also form a cross field of view. For another example, binocular cameras can also be installed at the front and rear ends of the moving direction of the mobile device, and a camera (such as a fisheye lens) can be installed on the upper and lower sides or the left and right sides of the mobile device, respectively. The two images are captured by the same camera at different times, but it is necessary to ensure that the two images have a common scene, that is, the speed of the mobile device needs to be combined to control the shooting time interval of the two images to ensure that the two images have a common scene; There are strict requirements for the same camera to shoot two images to calculate the depth map of obstacles. One is that the camera itself should move, and the other is that the captured obstacles need to be relatively static; in addition, a single camera shoots two images at different time points to calculate obstacles. The principle of the object depth map is equivalent to the binocular camera. At this time, it is necessary to calculate the displacement of the camera map when capturing images at different time points, which is equivalent to the distance between the two lenses of the binocular camera.

Step 2: Save at least two images as high-resolution images and low-resolution images, respectively.

High-resolution pictures are referred to as HD pictures (HD is the abbreviation of English High Definition), which refers to pictures with a vertical resolution greater than or equal to 720, also known as high-definition images, and the dimensions are generally 1280×720 and 1920×1080. In this embodiment, the high-resolution picture may adopt the above-mentioned standard or a relative standard, for example, it only needs to satisfy that the resolution of the high-resolution picture is greater than the resolution of the low-resolution picture. Specifically, in this embodiment, images captured by each camera of the binocular camera at the same time are obtained, two high-resolution pictures with a common scene are obtained and saved, and then the two high-resolution pictures are down-sampled respectively. , to obtain the corresponding low-resolution image and save it.

Step 3: Detect suspected obstacles based on low-resolution pictures of at least two images.

As shown in Figure 2, in this embodiment, this step includes the following sub-steps:

Sub-step 1: Acquire a region of interest in a low-resolution picture of at least two images.

The area of interest in this embodiment refers to the overlapping area of the fields of view of adjacent cameras. As shown in FIG. 3 , for adjacent cameras f1 and f3, the area of interest (ie, the ROI area) is the overlapping area S5 of the cameras f1 and f3 , the non-ROI area is the area other than S5; similarly, the area of interest of cameras f1 and f2 is S3; the area of interest of cameras f3 and f4 is S3'; the area of interest of cameras f2 and f4 is S6.

Sub-step 2: Perform stereo matching on the region of interest in the low-resolution pictures of at least two images.

A stereo matching process in this embodiment is as follows: Harris feature points of regions of interest in two low-resolution images are extracted respectively, and then matching feature points with higher accuracy are obtained through feature matching, thereby completing image stereo matching.

Sub-step 3: Calculate the depth map of the region of interest in the low-resolution pictures of at least two images.

In this embodiment, the pixel points corresponding to the matching feature points in sub-step 2 are obtained, and then the disparity information is calculated according to the triangulation principle, and then the disparity information is converted into a depth map through.

Sub-step 4: Calculate the obstacle distance according to the depth map of the region of interest.

Since the gray value of each pixel in the depth map can be used to represent the distance of obstacles in the scene, the distance value of obstacles can be obtained according to the depth map.

Sub-step 5: Determine whether there is a suspected obstacle according to the distance of the obstacle.

The obstacle distance value calculated according to sub-step 4 is compared with the preset threshold. If it is less than the threshold, it is considered that there is a suspected obstacle, and the process goes to step 4; if it is greater than the threshold, it is considered that there is no suspected obstacle. Then go back to step one.

Step 4: Obtain the detailed information of the suspected obstacle according to the high-resolution pictures of at least two images.

In this embodiment, when the distance value of the suspected obstacle is smaller than the threshold value, two corresponding high-resolution pictures are obtained, and then the regions of interest of the two high-resolution pictures are accurately matched and solved to obtain the interesting The depth map or point cloud of the suspected obstacle corresponding to the area is obtained, so as to obtain the detailed information of the suspected obstacle.

It can be known from the above steps that in this embodiment, the suspected obstacle is detected by using a low-resolution image, and the detailed information of the obstacle is obtained by performing accurate matching and solving through the high-resolution image after the suspected obstacle is found. This embodiment can effectively reduce information acquisition of irrelevant obstacles, and only acquire detailed information for suspected obstacles, which improves the efficiency and speed of acquiring obstacle information, and reduces the requirement for hardware (eg, a chip) to execute the method.

Embodiment 2:

As shown in FIG. 4 , it is a schematic diagram of a module of a mobile device in an embodiment of the present invention. The mobile device in this embodiment includes a mobile module, an image sensor, a processor, a memory, and is stored in the memory and can run on the processor. computer program. The mobile module can be the driving wheel of the mobile car or the propeller of the drone, which is used to drive the movement of the mobile device; the image sensor can be a binocular camera or a fisheye camera, which is used to obtain images around the mobile device. The cameras have crossed fields of view; the processor executes the computer program stored in the memory to implement the obstacle information acquisition method in Embodiment 1.

Embodiment three:

A computer-readable storage medium is disclosed in this embodiment, and a computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the method for obtaining obstacle information in Embodiment 1 is implemented.

Embodiment 4:

As shown in FIG. 5 , an obstacle avoidance method for a mobile device is disclosed in this embodiment, and the mobile device includes but is not limited to a mobile car and an unmanned aerial vehicle (UAV). The obstacle avoidance method of the mobile device of the present invention includes the following steps.

Step 1: Set a safe operating area centered on the mobile device.

As shown in Figure 6, the safe operation area of the mobile trolley is the safe operation pipeline shown by the dotted line in the figure. The distance R is a circular tubular channel of radius.

Step 2: Acquire at least two images of the same scene around the mobile device.

As shown in Figure 6, the images of the front or the side of the mobile trolley are acquired through the binocular cameras installed on the front and side of the mobile trolley.

Step 3: Save at least two images as high-resolution pictures and low-resolution pictures, respectively.

High-resolution pictures are referred to as HD pictures (HD is the abbreviation of English High Definition), which refers to pictures with a vertical resolution greater than or equal to 720, also known as high-definition images, and the size is generally 1280 × 720 and 1920 × 1080. In this embodiment, the high-resolution picture may adopt the above-mentioned standard or a relative standard, for example, it only needs to satisfy that the resolution of the high-resolution picture is greater than the resolution of the low-resolution picture. Specifically, in this embodiment, images captured by each camera of the binocular camera at the same time are obtained, two high-resolution pictures with a common scene are obtained and saved, and then the two high-resolution pictures are down-sampled respectively. , to obtain the corresponding low-resolution image and save it.

Step 4: Detect suspected obstacles based on low-resolution pictures of at least two images.

This step is basically the same as step 3 in Example 1, and this step includes the following substeps:

Sub-step 1: Obtain a region of interest in a low-resolution picture of at least two images.

Sub-step 2: Stereo matching the region of interest in the low-resolution pictures of at least two images.

Sub-step 3: Compute a depth map of the region of interest in the low-resolution picture of at least two images.

Sub-step 4: Calculate the obstacle distance based on the depth map of the region of interest.

Sub-step 5: Determine whether there is a suspected obstacle according to the obstacle distance.

According to the obstacle distance calculated in sub-step 4, judge whether the obstacle is located in the safety pipeline. If so, judge that the obstacle is a suspected obstacle, and then go to step 5; if not, judge that there is no suspected obstacle, and the interval Return to step 2 after a while.

Step 5: Obtain detailed information of suspected obstacles based on high-resolution pictures of at least two images.

When the obstacle is located in the safety pipeline, two high-resolution pictures of its corresponding images are obtained, and then the regions of interest of the two high-resolution pictures are accurately matched and solved to obtain the suspected obstacle corresponding to the region of interest. Depth map or point cloud to get detailed information about the suspected obstacle.

Step 6: Perform obstacle avoidance actions according to the obtained obstacle details.

Generate an environment map according to the obtained detailed information of suspected obstacles, and control the mobile car to perform obstacle avoidance actions, such as stopping, turning or retreating.

It can be seen from the above steps that in this embodiment, the low-resolution image is used to detect whether the obstacle is located in the safe operation area to determine whether there is a suspected obstacle, and the high-resolution image is used to accurately match and solve the problem after the suspected obstacle is found. to obtain detailed information on obstacles and perform obstacle avoidance actions. Because this embodiment only obtains detailed information for suspected obstacles, the efficiency and speed of obtaining obstacle information is improved, the speed of the drone or mobile car in responding to obstacles is improved, and the hardware (such as a chip) that executes the method is reduced. need.

Embodiment 5:

As shown in FIG. 6 , which is a schematic diagram of a mobile device in an embodiment of the present invention, the mobile device in this embodiment includes a mobile module, an image sensor, a processor, a memory, and a Computer program. The mobile module is the walking wheel of the mobile trolley, which is used to drive the mobile trolley to move; the image sensor is a binocular camera installed in the front or side of the mobile trolley, which is used to obtain the image of the front or side of the mobile trolley; the processor is located inside the mobile trolley (not shown in the figure), for executing the computer program stored in the memory (not shown in the figure) to implement the obstacle avoidance method of the mobile device in Embodiment 4.

Embodiment 6

A computer-readable storage medium is disclosed in this embodiment, and a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the obstacle avoidance method of the mobile device in Embodiment 4 is implemented.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium can include: Read only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, etc.

The above descriptions are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention shall be included in the protection of the present invention. within the range.

Claims

A method for obtaining obstacle information, comprising the following steps:

acquiring at least two images of the same scene around the mobile device;

Save at least two images as a high-resolution image and a low-resolution image;

Detect suspected obstacles based on low-resolution pictures of at least two images;

Obtain detailed information on suspected obstacles from high-resolution pictures of at least two images.
The method for obtaining obstacle information according to claim 1, wherein at least two images of the same scene around the mobile device are obtained at different times through a binocular camera or through the same camera.
The method for obtaining obstacle information according to claim 1, wherein the detecting a suspected obstacle according to a low-resolution picture of at least two images comprises:

Obtain a region of interest in a low-resolution picture of at least two images;

performing stereo matching on regions of interest in low-resolution pictures of at least two images;

calculating a depth map of a region of interest in a low-resolution picture of at least two images;

Calculate the obstacle distance based on the depth map of the region of interest;

Determine whether there is a suspected obstacle according to the distance of the obstacle.
A mobile device, comprising a mobile module, an image sensor, a processor, a memory, and a computer program stored in the memory and executable on the processor, the mobile module is used to drive the movement of the mobile device, and the image sensor is used to acquire The image, characterized in that the processor executes the computer program to implement the obstacle information acquisition method according to any one of claims 1 to 3.
A computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the obstacle information acquisition according to any one of claims 1 to 3 is realized method.
An obstacle avoidance method for a mobile device, characterized in that:

Set up a safe operating area centered on the mobile device;

acquiring at least two images of the same scene around the mobile device;

Save at least two images as a high-resolution image and a low-resolution image;

Detect suspected obstacles based on low-resolution pictures of at least two images;

Obtain detailed information of suspected obstacles based on high-resolution pictures of at least two images;

Perform obstacle avoidance actions based on the obtained obstacle details.
The obstacle avoidance method for a mobile device according to claim 6, wherein at least two images of the same scene around the mobile device are acquired at different times by a binocular camera or by the same camera.
The obstacle avoidance method for a mobile device according to claim 6, wherein the detecting a suspected obstacle according to low-resolution pictures of at least two images comprises:

Obtain a region of interest in a low-resolution picture of at least two images;

performing stereo matching on regions of interest in low-resolution pictures of at least two images;

calculating a depth map of a region of interest in a low-resolution picture of at least two images;

Calculate the obstacle distance based on the depth map of the region of interest;

Determine whether there is a suspected obstacle according to the distance of the obstacle.
A mobile device, comprising a mobile module, an image sensor, a processor, a memory, and a computer program stored on the memory and executable on the processor, the mobile module is used to drive the movement of the mobile device, and the image sensor is used to acquire The image, characterized in that the processor executes the computer program to implement the obstacle avoidance method for a mobile device according to any one of claims 6 to 8.
A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the avoidance of the mobile device according to any one of claims 6 to 8 is realized. obstacle method.