WO2018086050A1

WO2018086050A1 - Depth map generation method and unmanned aerial vehicle based on this method

Info

Publication number: WO2018086050A1
Application number: PCT/CN2016/105408
Authority: WO
Inventors: 周游; 朱振宇; 杜劼熹
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2016-11-11
Filing date: 2016-11-11
Publication date: 2018-05-17
Also published as: CN107077741A

Abstract

A method for generating a depth map capable of performing shielding recovery, comprising: S1, detecting abnormal areas in two images, wherein the two images are obtained by photographing the same scene from different angles at the same time; and S2, where pixels, located in the abnormal areas, in the two images are shielded, generating a depth map according to the two images. A shielding detection system, obstacle avoidance system and unmanned aerial vehicle based on the depth map generation method are capable of detecting and dealing with a fixed obstacle in real -time, thereby reducing the influence thereof on obstacle detection.

Description

Depth map generation method and drone based on the same

Copyright statement

The disclosure of this patent document contains material that is subject to copyright protection. This copyright is the property of the copyright holder. The copyright owner has no objection to the reproduction of the patent document or patent disclosure contained in the official records and files of the Patent and Trademark Office.

Technical field

The invention belongs to the field of computer vision technology, and specifically relates to a method for performing occlusion recovery when a binocular camera system generates a depth map for avoiding obstacles, and an occlusion detection system using the occlusion recovery method. The invention can be applied to obstacle avoidance applications of vehicles with multiple cameras, such as unmanned cars, self-propelled drones, VR/AR glasses, dual-camera mobile phones and the like.

Background technique

Computer vision is a technique that relies on an imaging system instead of a visual organ as an input-sensitive means. The most common imaging system is the camera, which consists of a dual camera that forms a basic vision system called Stereo Vision.

The binocular camera system Stereo Vision System uses two cameras to capture two photos at the same time and at different angles, and then through the difference between the two photos and the position and angle relationship between the two cameras, using the triangular relationship to calculate The distance between the scene and the camera. This distance relationship shows a depth map (Depth Map). That is to say, the binocular camera system acquires the depth information of the scene by the difference of two photos at different angles at the same time.

The difference between the two photos is usually caused by shooting from different angles, in which case the calculated scene depth information is correct. However, this difference may also be caused by imaging differences between the two cameras themselves, or by a single camera being occluded. If this is the case, then the calculated depth information is wrong.

In order to eliminate the imaging difference between the two cameras caused by the occlusion of a single camera, it is necessary to eliminate the influence of the occlusion, that is, perform an occlusion recovery operation. When the binocular camera system is applied to the obstacle avoidance application of unmanned vehicles such as drones, if the calculation of the depth map is wrong, it will cause false detection of the obstacle, thereby erroneously triggering the braking measure, thereby affecting no The operation and user experience of the vehicle.

Summary of the invention

The invention aims to solve the problem that the fixed obstruction in the binocular image affects the depth map calculation when the UAV is obstacle avoidance, thereby causing the obstacle avoidance error.

In order to solve the above technical problem, the present invention provides a depth map generation method capable of performing occlusion recovery, comprising: S1, detecting an abnormal region in two images, which are obtained by simultaneously shooting the same scene from different angles. S2, in the case of masking pixels in the abnormal region among the two images, generating a depth map according to the two images.

The present invention also provides a drone, comprising: an image acquisition device for acquiring two images simultaneously photographing the same scene from different angles; and one or more processors for detecting the two images An abnormal region in the middle; and in the case of masking the abnormal region, a depth map is generated from the two images.

The present invention also proposes a processor for executing the above-described depth map generation method, and a computer readable medium storing a program for executing the above-described depth map generation method.

The invention proposes a method for detecting occlusion on a depth map and removing its influence in real time from an engineering point of view. The method of the invention is simple and effective, has strong robustness, can be detected and processed in real time, and reduces the rate of false detection of obstacles.

DRAWINGS

1A is a schematic diagram of an image taken from one of the front binocular cameras of the drone in one embodiment of the present invention;

1B is a schematic diagram of performing abnormal region detection in an embodiment of the present invention;

2 is a flow chart of steps included in the depth map generation method of the present invention;

3 is a schematic diagram of a depth map of FIG. 1A generated by a depth map generation method in accordance with the present invention;

4 is a block diagram showing an embodiment of an occlusion detecting system for a drone according to the depth map generating method architecture of the present invention;

5 is a block diagram of a module of an obstacle avoidance system of a drone according to an embodiment of the present invention;

Figure 6 is a block diagram of a module of a drone according to another embodiment of the present invention.

detailed description

Since the depth map can display the distance of the object in the scene, it is possible to detect whether there is an obstacle in the forward direction of the drone according to the depth map, thereby performing corresponding obstacle avoidance operation accordingly. However, in practical applications, the occlusion of the components of the drone itself is often present in the camera, such as a propeller of a drone, a propeller cover, and the like. When calculating the depth map, if parts of these parts appear in the camera, they are often treated as obstacles that are close together, causing misoperation. To this end, the present invention proposes to eliminate the interference of such occlusions when calculating the depth map, that is, to recover the occlusion.

In general, the present invention proposes a depth map generation method for occlusion recovery, and a drone using the generation method. The method of generating the depth map includes two basic steps: S1, detecting an abnormal region in the two images; S2, generating a depth map in the case of masking pixels in the abnormal region among the two images. Of course, the two images here are images taken by a binocular camera, that is, the two images are obtained by simultaneously shooting the same scene from different angles.

That is, one aspect of the present invention is to detect abnormal areas that may be considered as obstacles, and on the other hand, how to make these abnormal areas not be regarded as obstacles. The basic principle of the present invention is to make the abnormal region not affect the calculation of the entire image depth map when generating the depth map. However, it should be understood that the present invention is not limited to how to calculate a depth map, but rather, the present invention is not included in the detected obstacle region not participating in the depth map calculation, or participating in the depth map calculation in a less-participating manner. The scope covered.

According to the present invention, it is possible to formulate a specific one according to the characteristics of an abnormal region appearing in an actual application. The abnormal region detecting method, the present invention is not limited to a specific detecting method, and an abnormal region in one image can be obtained after the detecting. For example, the detection may be performed by detecting the texture of the occlusion that may occur, the area where the occlusion may occur, and the like.

In addition to the occlusion of the UAV's own components, scenes such as the sky, strong light sources, and the sun that cause image overexposure can also affect the generation of depth maps. These overexposed areas may be considered as areas where obstacles appear. The invention also proposes that when detecting an abnormal region in two images, the overexposed region is also used as an abnormal region to eliminate the influence of these regions.

It can be seen from the above that the present invention improves the existing depth map generation method to minimize the influence on the obstruction. In this way, the drone with occlusion recovery of the present invention can be designed.

Furthermore, the present invention also proposes to include a drone having an occlusion recovery function, which should also include an image acquisition device for acquiring two images simultaneously photographing the same scene from different angles. And one or more processors for detecting an abnormal region in the two images; and in the case of masking the abnormal region, generating a depth map according to the two images. Thereby, the drone performs occlusion detection based on the image acquired by the image acquisition device.

The following description will be made with reference to specific embodiments. The following description is directed to the use of a drone as an example. However, the method, system, and apparatus of the present invention are not limited to the field, and any application field that requires calculation of a depth map can reasonably utilize the present invention, for example, including any Various application devices and application scenarios that require humans to operate directly on the device body, such as robot technology, virtual reality technology, and the like.

1A is a schematic illustration of an image taken from a camera in a front-mounted binocular camera of a drone in one embodiment of the present invention. As shown in Fig. 1A, a long strip of object A appears in the upper right of the picture, which is the blade guard of the drone. The pad guard usually appears in a fixed area of the camera to the range of the shot. In addition, the propeller, tripod and other components of the drone may appear in the image, and the position is usually fixed. In this embodiment, the paddle protective cover shown in FIG. 1 will be described as an example.

There are vertical stripes on the blade protection cover. Therefore, before the drone takes off, the fixed area where the blade guard may appear is tested. As shown in Fig. 1B, the fixed area B is detected, in this embodiment, a texture detection is performed on a known occlusion to determine whether occlusion occurs. Here, the present invention first performs binarization processing, turns black and white, and then counts the pixel data of one line according to the direction of the arrow C, and uses 0 to indicate white and 1 to represent black. The following law [000000001111 000000111 000011 001] should be present: 1 Black and white phase 2 In the left picture, the black band and the leucorrhea are gradually reduced from left to right, and in the right picture, the number of repetitions of black and white is 9 times, and the number of times is more than 8 times. If the above three conditions are met, it can be considered that the blade protection cover is detected, and the window prompt can be further displayed on the interactive interface such as the mobile phone APP, and the user is required to confirm whether the blade protection cover is actually installed. Through such a procedure, the occlusion of the blade guard can be quickly confirmed, thereby triggering the corresponding occlusion recovery logic.

It should be understood that the above method for detecting texture is merely an example, and the present invention can formulate a specific detection method according to the existing method according to the specific shape, pattern, and the like of the specific occlusion, which should be covered by the present invention. Within the scope.

Furthermore, the present invention also contemplates that the position of the component itself appearing in the image may not be fixed relative to the image acquisition device (binocular camera). For example, in the process of flying a drone or before the drone takes off, in order to make the shooting device fixed on the gimbal (unlike the binocular camera) continue to shoot a certain scene, the attitude of the gimbal may be at any time. Changed. Thus, the area, size, or shape that the gimbal may appear in the image acquired by the binocular camera may vary. In this case, it is necessary to judge whether there is an abnormal area in the image according to the posture of the gimbal, and perform corresponding occlusion detection and depth map occlusion recovery operation. Specifically, the position of the gimbal in the detected image can be estimated according to the real-time attitude angle of the gimbal (such as the pitch angle, the roll angle, the yaw angle, etc.), and the occlusion recovery operation is performed after estimating the position of the gimbal. .

Further, according to another embodiment of the present invention, when detecting an abnormal region in two images, an overexposed region is regarded as an abnormal region. The overexposed area includes a strong light area generated by direct illumination such as sunlight, light, or the like, or may be an overexposed area due to high dynamic scene switching. Since the overexposed area affects the calculation of the depth map like the occlusion, these overexposed areas are also treated as a special abnormal area when detecting the abnormal area. The area where the overexposure is detected may be performed by detecting the luminance value of each pixel in the image, for example, an area in which the luminance value in the image is larger than a certain threshold may be used as the overexposed area. Of course, other techniques for detecting overexposed regions can be employed with the present invention. In a In some embodiments, the exposure time of the image acquisition device is set to be very short (eg, less than 5 ms), and the gain thereof is also small, but many pixel values appear to be larger than the threshold (if there are a large number of pixels with pixel values greater than 255) The pixel point at this time can be judged as overexposed.

2 is a schematic diagram of the depth map of FIG. 1A generated by a depth map generation method in accordance with the present invention. As shown in Figure 2, the generated depth map is unaffected by the obstruction. 3 is a flow chart showing the steps included in the depth map generation method of the present invention. As shown in Figure 3, the method of generating the depth map includes two basic steps:

S1: detecting an abnormal region in two images obtained by simultaneously shooting the same scene from different angles;

S2. In the case of masking pixels in the abnormal region among the two images, a depth map is generated according to the two images.

In the above-described embodiment of the present invention, the abnormal region in the two images is detected by the above-described detecting method, and then the depth map is generated in the case of masking the pixels of the abnormal region. When performing the masking process, the present invention may adopt different shielding methods according to a specific depth map generation method, and the principle is that the pixels of the abnormal region do not participate in the calculation of the depth map, or the components participating in the depth map are involved. The calculation does not have a significant impact.

As an example, when generating the depth map D in step S2, a Semi-global Matching algorithm may be used, which calculates a corresponding depth value d when the matching cost S takes a minimum value, and S represents a matching of the two images when performing matching calculation. degree. The matching cost S is calculated by:

Where L _r (p,d) represents the path cost of the pixel p along the path r, and the path cost L _r (p,d) is obtained by iterative calculation of:

L' _r (p,d)=C(p,d)+min(L' _r (pr,d), L' _r (pr,d-1)

+P ₁ , L' _r (pr,d+1)+P ₁ ,min(L' _r (pr,i)+P ₂ )

Where C(p,d) represents the matching degree of the pixel p, P ₁ and P ₂ are penalty factors, all of which are constants, and i is a natural number.

All pixels of all of the abnormal region detected in step 1 to make C (p, d) is a constant C _B, C _B is greater than that of the other region of the pixel C (p, d) maximum.

According to the above formula, since L' _r (p,d) depends only on the latter part of the formula, the depth information of this point is estimated by the surrounding points, so that the problem of matching errors of the occlusion reference itself can be removed.

According to an embodiment of the present invention, when the abnormal area is detected in step S1, step S2 further includes the step of prompting the occurrence of occlusion. For example, for the Semi-global Matching algorithm to calculate the depth map, ∑ _p S(p,d) can also be calculated. When ∑ _p S(p,d) is greater than a threshold, occlusion is prompted. Since the method manually sets the matching result of some pixels to make C(p,d) a large number, it will cause a large deviation, so when calculating the adaptation parameters of the entire matching process, manually set Pixels will never participate in the calculation. Therefore, if the calculated ∑ _p S(p,d) is large (greater than a certain threshold), the current observation is considered to be poor, object occlusion may occur, or other conditions (the lens is contaminated Etc.), resulting in poor matching between binoculars. At this time, the drone can send a prompt message to the user, such as issuing a warning message on the control end (such as a remote controller or a smart phone), warning the user that there may be obstructions blocking the image acquisition device. After the user receives the warning message, they can check to see if there is an obstruction.

4 is a block diagram showing an embodiment of an occlusion detecting system for a drone according to the depth map generating method architecture of the present invention. As shown in FIG. 4, the detection system 1 includes a depth map generating device 10 and an occlusion detecting device 10 for detecting an abnormal region in two images simultaneously photographing the same scene from different angles, and In the case of masking the pixels in the abnormal region among the two images, a depth map is generated according to the two images; the occlusion detecting device 11 is configured to detect the occlusion according to the depth map generated by the depth map generating device.

The depth map generating device 10 uses the foregoing depth map generating method of the present invention, and details are not described herein again. The occlusion detecting device 11 determines whether or not there is an obstruction based on the generated depth map. For example, for the Semi-global Matching algorithm to calculate the depth map, as described above, ∑ _p S(p,d) can be calculated, and when ∑ _p S(p,d) is greater than a threshold, occlusion is determined.

As shown in FIG. 4, the occlusion detection system of the present invention may further include an alerting device 12. When the occlusion detecting device 11 detects that there is an obstruction, it sends an alert instruction to the alert device 12, The warning device performs an alerting action when the occlusion detecting device determines that occlusion occurs due to the warning command.

As a specific implementation of the application on the UAV, the warning device 12 can be any device that can be used to generate sound, light, and electric signals, such as a speaker, a warning light, or the like, mounted on the drone. In addition to this, the present invention is preferably a device in which the alert device 12 is designed to have a human-computer interaction function, or is integrated in a device having a human-computer interaction function. For drones, it can be placed on the remote control of the drone, or by existing devices in the remote control of the drone, such as screens, lights, and the like.

As a more preferred embodiment, the alerting device is a device that can receive user input. For example, the alert device can be a touch screen, or a combination of a screen and a button, and the like. When the alert device 12 receives the alert command, it can alert the user by displaying a pop-up window on the screen and prompt the user to confirm whether occlusion is indeed present. Thereby, the user can directly perform the relevant confirmation action.

Figure 5 is a block diagram of a block diagram of an obstacle avoidance system for a drone in accordance with one embodiment of the present invention. As shown in FIG. 5, the obstacle avoidance system includes an image acquisition device 2, an occlusion detection system 1, and an obstacle avoidance control device 3.

The image acquisition device 2 is for acquiring two images that simultaneously capture the same scene from different angles. Usually the image acquisition device is a binocular camera, and the binocular camera may be a visible light band camera or an infrared camera, or both. The image acquisition device 2 transmits the acquired image frame in the picture or video to the occlusion detection system.

The occlusion detection system 1 can generate a depth according to the two images and perform occlusion detection. The specific implementation has been described in the foregoing, and will not be described herein.

The obstacle avoidance control device 3 is configured to perform obstacle avoidance control for an unmanned vehicle such as a drone, and can automatically control the traveling or flight of the unmanned vehicle. It typically has its own obstacle detection device and adjusts the travel or flight path of the driverless vehicle based on the detected obstacle. In the present invention, in order to prevent the obstruction from being regarded as an obstacle, the present invention introduces the aforementioned occlusion detection system 2, and causes the obstacle avoidance control device 3 to use the detection result of the occlusion detection system 2 as a basis for whether or not to perform obstacle avoidance. one. Specifically, when the occlusion detecting device detects that there is occlusion, the obstacle avoidance control device 3 does not perform obstacle avoidance on the image abnormality generated by the occlusion object, and When the occlusion detecting device detects that there is no occlusion, the obstacle avoidance control device 3 performs obstacle avoidance in a conventional manner.

In practical applications, the occlusion detection system and the obstacle avoidance control device may be implemented by independent hardware or software, or may be implemented by integrated hardware or software. For drones, they can also be implemented together as hardware and/or software as part of a flight control system.

Figure 6 is a block diagram of a module of a drone according to another embodiment of the present invention. As shown in FIG. 6, the drone includes an image acquisition device 2 and a processor 5. The image acquisition device 2 is configured to acquire two images that simultaneously capture the same scene from different angles; the processor 5 is configured to detect an abnormal region in the two images, and in the case of shielding the abnormal region, according to the The two images are generated to generate a depth map. The processor 5 has the foregoing method for generating a depth map, and details are not described herein again.

The processor 5 can also determine whether there is a large area occlusion according to the generated depth map. For example, when calculating the depth map for the Semi-global Matching algorithm, as described above, ∑ _p S(p,d) can be calculated. When ∑ _p S(p,d) is greater than a threshold, it is judged that large-area occlusion occurs. .

As shown in FIG. 6, the drone of this embodiment may further include a display device 6, which may also cause the display device 6 to display the warning information when the occlusion occurs. The display device may, for example, be a touch screen capable of receiving user input to enable a user to over-enter to confirm whether there is a large area of occlusion.

It should be understood that although one processor is employed in this embodiment, it is also possible for a plurality of processors to perform the steps performed in the one processor described above, respectively.

The above method and module according to various embodiments of the present invention may be implemented by executing a computer-containing software by a computing-capable electronic device. The system can include storage devices to implement the various storages described above. The computing capable electronic device can include a general purpose processor, a digital signal processor, a dedicated processor, a reconfigurable processor, etc., but is not limited thereto. Executing such instructions causes the electronic device to be configured to perform the operations described above in accordance with the present invention. The above methods and/or modules may be implemented in one electronic device or in different electronic devices.

Embodiments of the invention use software, which may be stored in the form of a volatile memory or a non-volatile storage device (such as a storage device such as a ROM), whether erasable or not. Still rewritable, or stored in the form of a memory (eg, RAM, memory chip, device, or integrated circuit), or stored on an optically readable or magnetically readable medium (eg, CD, DVD, disk, tape, etc.) Wait). It should be appreciated that the storage device and the storage medium are embodiments of a machine-readable storage device adapted to store one or more programs, the program or programs comprising instructions that, when executed, implement the present invention An embodiment. Moreover, these programs can be routed via any medium, such as a communication signal carried via a wired connection or a wireless connection, and various embodiments suitably include such programs.

The specific embodiments of the present invention have been described in detail in the foregoing detailed description of the embodiments of the present invention. All modifications, equivalents, improvements, etc., made within the spirit and scope of the invention are intended to be included within the scope of the invention.

Claims

A depth map generation method capable of occlusion recovery, comprising:

S1: detecting an abnormal region in two images obtained by simultaneously shooting the same scene from different angles;

S2. In the case of masking pixels in the abnormal region among the two images, a depth map is generated according to the two images.
The depth map generating method according to claim 1, wherein detecting the abnormal region in the two images in step S1 is performed by detecting a texture of the occlusion and/or an area where the occlusion may appear.
The depth map generating method according to claim 2, wherein an area where the occlusion object may appear is determined according to an attitude angle of the occlusion.
The depth map generating method according to claim 1, wherein when the abnormal region in the two images is detected in step S1, the overexposed region is regarded as an abnormal region.
The depth map generating method according to claim 4, wherein the overexposed area refers to an area in which the luminance value of the pixel is greater than a certain threshold.
The depth map generating method according to claim 1, wherein the depth map D is generated in step S2 by using a Semi-global Matching algorithm, and the algorithm calculates a corresponding depth value d when the matching cost S takes a minimum value, and S represents the two The degree to which the image matches when performing a matching calculation.
The depth map generating method according to claim 6, wherein

The matching cost S is calculated by:

Where L r (p,d) represents the path cost of the pixel p along the path r, and the path cost L r (p,d) is obtained by iterative calculation of:

L' r (p,d)=C(p,d)+min(L' r (pr,d), L' r (pr,d-1)+P 1 ,L' r (pr,d+1 ) + P 1 ,min(L' r (pr,i)+P 2 )

Where C(p,d) represents the matching degree of the pixel p, P 1 and P 2 are penalty factors, all of which are constants, and i is a natural number.
The depth map generating method according to claim 7, wherein

In step S2, all the pixels of all of the abnormal region detected in step 1 to make C (p, d) is a constant C B, C B is greater than that of the other region of the pixel C (p, d) maximum.
The depth map generating method according to claim 8, wherein the step S2 further comprises the step of prompting the occurrence of large-area occlusion.
The depth map generating method according to claim 9, wherein step S2 calculates ∑ p S(p, d), and when ∑ p S(p, d) is greater than a threshold, it indicates that large-area occlusion occurs.
A drone, characterized in that it comprises:

An image acquisition device for acquiring two images simultaneously photographing the same scene from different angles;

One or more processors for:

Detecting an abnormal region in the two images;

In the case of masking the abnormal region, a depth map is generated from the two images.
The drone of claim 11 wherein said one or more processors are operable to detect anomalous regions in said two images by detecting texture of the obstruction and/or an area in which the obstruction may occur.
The drone according to claim 12, wherein an area in which said covering object may appear is determined based on an attitude angle of the obstructing object.
The drone according to claim 11, wherein the one or more processors detect an abnormal region in the two images as an abnormal region.
The drone according to claim 14, wherein the overexposed area is an area in which the luminance value of the pixel is greater than a certain threshold.
The drone according to claim 11, wherein said one or more processors generate a depth map D using a Semi-global Matching algorithm, and the algorithm calculates a corresponding depth value d when the matching cost S takes a minimum value, and S represents The degree of matching of the two images when performing matching calculation.
The drone of claim 16 wherein

The matching cost S is calculated by:

Where L r (p,d) represents the path cost of the pixel p along the path r, and the path cost L r (p,d) is obtained by iterative calculation of:

L' r (p,d)=C(p,d)+min(L' r (pr,d), L' r (pr,d-1)+P 1 ,L' r (pr,d+1 ) + P 1 ,min(L' r (pr,i)+P 2 )

Where C(p,d) represents the matching degree of the pixel p, P 1 and P 2 are penalty factors, all of which are constants, and i is a natural number.
The drone according to claim 17, wherein

For all the pixels of all of the abnormal region is detected, so that C (p, d) is a constant C B, C B is greater than that of the other region of the pixel C (p, d) maximum.
The drone of claim 18, wherein said one or more processors are operative to calculate ∑ p S(p,d), and when ∑ p S(p,d) is greater than a threshold, determining a large area Occlusion.
The drone of claim 19, the one or more processors further configured to issue a warning message when the occlusion occurs.
The drone of claim 20, further comprising display means for causing the display means to display the warning message when the occlusion occurs.
The drone of claim 21, said display device being capable of receiving user input to enable a user to pass an input to confirm whether there is a large area of occlusion.
A processor for performing the depth map generation method according to any one of claims 1 to 10.
A computer readable medium storing a program for performing the depth map generating method according to any one of claims 1 to 10.