KR20200008409A - A method and apparatus for detecting unmanned aerial vehicle using stereo camera and additional camera - Google Patents

Abstract

The present invention relates to a method and an apparatus for detecting obstacles of an unmanned vehicle. More specifically, the present invention has a sub camera between a pair of stereo cameras, allows an image photographed by the sub camera to be changed based on an obstacle event, and detects obstacles based on the image photographed by the sub camera and the image of the pair of stereo cameras.

Description

A method and apparatus for detecting unmanned aerial vehicle using stereo camera and additional camera}

The present invention relates to an obstacle detection system and method for detecting an obstacle for autonomous flight of an unmanned aerial vehicle, and additionally adding an additional camera capable of physical control to a stereo camera.

As drones are gradually developed, drones are being developed for autonomous flight and specific tasks performed by the drone to determine the surrounding environment in a manual control mode using a remote controller. To this end, technologies for detecting and avoiding obstacles in the flight path are actively researched to prevent collision of the drone during autonomous flight. Currently, obstacle detection of drones uses a single or combined method of various sensors such as a stereo camera, a lidar, and an ultrasonic wave.

In particular, the obstacle detection system based on a stereo camera is applicable to most drone obstacle avoidance systems because an obstacle avoidance system can be constructed at a lower price than an expensive lidar.

However, the stereo camera-based obstacle detection system has a limitation in the detection distance according to the baseline length, which is the distance between the stereo cameras. In this case, if the baseline length between cameras is increased for the long range detection, the long distance obstacle can be detected, but in the case of the short range obstacle, the recognition rate may decrease due to occlusion or parallax error. In addition, since stereo cameras basically measure the distance of objects using parallax of horizontal lines, there is a disadvantage in that the recognition rate is very low in a long thin structure such as an electric wire or a clothesline.

In addition, the drone is limited to mounting a large number of sensors due to the limitations of loading weight, battery, and HW resources, and thus, a drone cannot be attached to all sensors having a desired function.

Therefore, in order to detect obstacles for autonomous flight of the drone, a technique for efficiently detecting obstacles based on a stereo camera is required.

An object of the present invention is to perform obstacle detection required for the safe operation of the drone.

An object of the present invention is to provide a method for performing obstacle detection by overcoming the constraints of weight and HW resources in an unmanned aerial vehicle.

An object of the present invention is to propose a method for solving a detection problem that is difficult to solve in the conventional stereo obstacle detection method in the drone.

An object of the present invention is to provide a method for detecting obstacles in the near and far distance in a drone obstacle detection system based on a stereo camera.

An object of the present invention is to overcome the binocular parallax error in a drone obstacle detection system based on a stereo camera.

An object of the present invention is to overcome a tilt phenomenon error in a drone obstacle detection system based on a stereo camera.

In addition, another object of the present invention is to provide a method for detecting a horizontal structure, such as a wire difficult to detect obstacles in the conventional stereo system.

An object of the present invention is to determine the obstacle in the unmanned aerial vehicle obstacle detection system based on the stereo camera and to register it to correct the image.

Technical problems to be achieved in the present invention are not limited to the above-mentioned technical problems, and other technical problems not mentioned above will be clearly understood by those skilled in the art from the following description. Could be.

According to one embodiment of the present invention, a drone may provide a method for detecting an obstacle. In this case, the method for detecting an obstacle by the drone includes receiving an image from a pair of stereo cameras, generating a depth map from the received image, and analyzing the depth map to determine whether a failure event occurs. .

At this time, there is a sub camera between the pair of stereo cameras, the image captured by the sub camera is changed based on the fault event, the image captured by the sub camera, and the image of the pair of stereo cameras. Obstacles can be detected based on.

In addition, the following embodiments may be applied in common in a method and apparatus for detecting an obstacle by a drone.

According to an embodiment of the present invention, when comparing images received from a pair of stereo cameras, it is determined whether there is a point or region matching between images, but if there is no matching point or region between the images You can register as an event. In this case, when a failure event is registered, the sub camera may be arranged such that the baseline between the driving shaft of the sub camera and the stereo camera is parallel to each other. In this case, stereo matching may be performed with one of the left and right images acquired by the sub camera and the stereo camera.

According to an embodiment of the present invention, by analyzing the generated depth map, it is determined whether the first region in the image is greater than or equal to the threshold, and when it is determined that the first region is greater than or equal to the threshold in the image, the disorder event. It can be registered as. When the obstacle event is registered, the inclination angle of the drone may be determined, and the first angle may be determined according to the determined angle. In addition, the sub-camera may capture an image that is different from the input image of the stereo camera by a first angle. At this time, the overlapping range between the image obtained from the sub-camera and the image of the stereo camera can be identified, the depth value of the overlapping range can be measured, and the depth value of the overlapping range can be expanded.

When analyzing the generated depth map according to an embodiment of the present invention, it is determined whether there is a horizontal line in the image, but if it is determined that there is a horizontal line in the image, there are two or more horizontal lines in the image. In this case, the failure event may be registered. When registered as the fault event, the sub camera may be arranged such that the base line between the driving shaft of the sub camera and the stereo camera is perpendicular to each other. In addition, vertical stereo matching between the vertically disposed sub-camera and the stereo camera may be performed to finally determine whether the horizontal line in the image is an actual obstacle.

According to one embodiment of the present invention, a drone may provide an apparatus for detecting an obstacle. In this case, the apparatus for detecting an obstacle by the drone includes an image acquisition unit that receives an image from a pair of stereo cameras, a depth map generator that generates a depth map from the input image, and analyzes the depth map to determine whether a failure event occurs. It may include a depth map image analysis unit and a sub-camera control unit for determining the.

In this case, a sub camera may exist between the pair of stereo cameras. In addition, an image captured by the sub camera may be changed based on the fault event, and an obstacle may be detected based on an image captured by the sub camera and an image of the pair of stereo cameras.

According to the present invention can provide a method for performing obstacle detection required for the safe operation of the drone.

According to the present invention, it is possible to provide a method for performing obstacle detection by overcoming constraints of weight and HW resources in an unmanned aerial vehicle.

According to the present invention, a method for solving a detection problem that is difficult to solve in the conventional stereo obstacle detection method in an unmanned aerial vehicle can be proposed.

According to the present invention, it is possible to provide a method for detecting near and far obstacles in a drone obstacle detection system based on a stereo camera.

According to the present invention, a method for overcoming binocular parallax error in a drone obstacle detection system based on a stereo camera can be provided.

According to the present invention, a method for overcoming a tilt phenomenon error in a drone obstacle detection system based on a stereo camera can be provided.

According to the present invention, it is possible to provide a method for detecting a horizontal structure such as a wire, which is difficult to detect an obstacle of a conventional stereo system.

According to the present invention, in a stereo camera-based drone obstacle detection system, an obstacle may be determined and registered to correct an image.

Effects obtained in the present invention are not limited to the above-mentioned effects, and other effects not mentioned above may be clearly understood by those skilled in the art from the following description. will be.

1 is a block diagram of an obstacle detection system using a plurality of cameras according to an embodiment of the present invention.
2 is a block diagram of a depth map image analyzing unit in a plurality of camera-based drone obstacle detection systems according to an exemplary embodiment of the present invention.
3 is a block diagram of a sub-camera control and recognition unit in a multi-camera drone obstacle detection system according to an exemplary embodiment of the present invention.
4 shows an example of a camera device configuration of a multi-camera-based drone obstacle detection system according to an embodiment of the present invention.
5 is a diagram illustrating a binocular parallax error and an error correction method according to an embodiment of the present invention.
FIG. 6 is a diagram illustrating an example of a tilt phenomenon that occurs when a drone moves at a high speed according to an embodiment of the present invention.
7 is a diagram illustrating an example of a method of determining a horizontal obstacle according to an embodiment of the present invention.
8 is a flowchart illustrating an obstacle detection method according to an embodiment of the present invention.
9 is a flowchart illustrating a method for detecting an obstacle when a parallax error occurs according to an embodiment of the present invention.
10 is a flowchart illustrating a method for detecting an obstacle when a tilt phenomenon occurs in an embodiment of the present invention.
FIG. 11 is a flowchart illustrating an obstacle detecting method when a horizontal obstacle is found according to an embodiment of the present invention.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention.

In the following description of embodiments of the present invention, when it is determined that detailed descriptions of well-known structures or functions may obscure the gist of the present invention, detailed descriptions thereof will be omitted. In the drawings, parts irrelevant to the description of the present invention are omitted, and like reference numerals designate like parts.

In the present invention, when a component is "connected", "coupled" or "connected" with another component, it is not only a direct connection, but also an indirect connection in which another component exists in between. It may also include. In addition, when a component "includes" or "having" another component, it means that it may further include another component, without excluding the other component unless otherwise stated. .

In the present invention, the terms "first" and "second" are used only for the purpose of distinguishing one component from other components, and do not limit the order or the importance between the components unless otherwise specified. Therefore, within the scope of the present invention, a first component in one embodiment may be referred to as a second component in another embodiment, and likewise, a second component in one embodiment may be referred to as a first component in another embodiment. It may also be called.

In the present invention, the components distinguished from each other to clearly describe each feature, and does not necessarily mean that the components are separated. That is, a plurality of components may be integrated into one hardware or software unit, or one component may be distributed into a plurality of hardware or software units. Therefore, even if not mentioned otherwise, such integrated or distributed embodiments are included in the scope of the present invention.

In the present disclosure, components described in various embodiments are not necessarily required components, and some may be optional components. Therefore, an embodiment consisting of a subset of the components described in an embodiment is also included in the scope of the present invention. In addition, embodiments including other components in addition to the components described in the various embodiments are included in the scope of the present invention.

The present invention relates to an obstacle detection system and method in which an additional camera capable of physical control is further added to a stereo camera for obstacle detection for autonomous flight of a drone. More specifically, the present invention relates to a drone obstacle detection system and method for performing a short distance obstacle and a horizontal obstacle detection that is difficult to detect with a conventional stereo system by using a motor or mechanically controlled additional camera in a stereo camera.

Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments set forth below, but may be implemented in various forms, and only the present embodiments are intended to complete the disclosure of the present invention, and those skilled in the art to which the present invention pertains. It is provided to fully inform the person having the scope of the invention, which is defined only by the scope of the claims.

Hereinafter, the present invention will be described with reference to the drawings of a block diagram or a processing flowchart for explaining a drone obstacle detection system and a method thereof according to embodiments of the present invention. It will be appreciated that each block of the flowchart illustrations and combinations of flowchart illustrations may be performed by computer program instructions. Since these computer program instructions may be mounted on a processor of a general purpose computer, special purpose computer, or other programmable data processing equipment, those instructions executed through the processor of the computer or other programmable data processing equipment may be described in the flowchart block (s). It will create means to perform the functions.

Each block may also represent a module, segment or portion of code that includes one or more executable instructions for executing a specified logical function (s). It should also be noted that in some alternative implementations, the functions noted in the blocks may occur out of order.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of an obstacle detection system using a plurality of cameras according to an embodiment of the present invention.

The drone detection system 100 according to an exemplary embodiment of the present invention includes an image acquisition unit 120, a depth map generator 130, a depth map image analyzer 140, a sub camera control and recognition unit 150, and an obstacle. The recognition unit 160 may be included.

The image acquirer 120 may include a pair of stereo cameras and an additional sub camera. Although one pair of stereo cameras and one sub camera are specified as an embodiment of the present invention, the number of stereo cameras and sub cameras may be one or more according to the application purpose and method of the present invention. It is not limited to the example.

The image acquisition unit 120 acquires a plurality of input images synchronized from the outside using a pair of stereo cameras and an additional sub camera. In more detail, the image acquisition unit 120 may acquire a plurality of input images at the same time according to the clock synchronization signal.

The stereo camera of the image acquisition unit 120 may be configured to be aligned in a horizontal line. In this case, the pair of stereo cameras may be separated by a predetermined interval, and the pair of stereo cameras may be configured to lie in a straight line.

In one embodiment of the present invention, the spacing between a pair of stereo cameras may be configured to have a baseline of sufficient length to detect distant obstacles in a fast moving drone.

In addition, the stereo camera of the image acquisition unit 120 may acquire a plurality of stereo images having binocular parallax.

The sub-camera of the image acquisition unit 120 may be set to a normal mode at the initial operation of the device. In this case, the sub camera set in the normal mode may be located between the baselines of the pair of stereo cameras. At this time, in one embodiment of the present invention, the sub-camera may be located in the middle between a pair of stereo cameras, but is not limited to the above-described embodiment. Also, the sub camera may be disposed on the base line of the pair of stereo cameras.

At this time, in one embodiment of the present invention, the driving shaft to which the sub-camera and the driving motor are connected may be parallel to a straight line formed by the pair of stereo cameras. In this case, the sub camera in the normal mode may be used to detect near obstacles that are difficult to be grasped by the stereo camera. Alternatively, in normal mode, a pair of stereo cameras can be used to detect distant obstacles.

The depth map generator 130 performs a function of extracting a depth map using a stereo image. Two cameras arranged in a horizontal line may acquire a plurality of images having different parallaxes according to the distance from the camera to the object. The depth map generator 130 performs stereo matching from the obtained plurality of images. In more detail, the stereo matching process finds the same position in two images, calculates parallax between image objects, and acquires a depth map using the parallax.

The depth map image analyzing unit 140 analyzes the generated depth map, classifies objects, extracts obstacle candidates present in the moving direction of the drone, and stores a corresponding result as an event.

The depth map image analyzing unit 140 analyzes the generated depth map to detect a parallax error with respect to a short range obstacle that cannot be recognized due to a long baseline and to store the result as an event.

The depth map image analyzing unit 140 analyzes the generated depth map to detect a tilt phenomenon caused by tilting the drone when the drone proceeds at a high speed, and stores the result as an event. At this time, in one embodiment of the present invention, the detection of the tilt phenomenon may use a method of analyzing the IMU information of the drone, a method of analyzing the horizontal line of the image, and the like.

The depth map image analyzer 140 analyzes the generated depth map and checks whether a horizontal line exists in the flight depth map to check whether there is a horizontal obstacle such as an electric wire that is a great risk for the flight of the drone. At this time, in one embodiment of the present invention, the horizontal line detection may be performed by extracting the horizontal line edge of the image.

At this time, the horizontal line can be detected such as the horizon or the actual wire. However, simple horizontal line detection alone cannot determine whether this line is a horizontal line that is an obstacle to the actual drone flight. More specifically, since it is not possible to determine whether the detected horizontal line is a wire that becomes an obstacle of the drone flight or a horizon rather than an obstacle, the sub camera control and recognition unit 150 stores the detected result as an event to check it. Do this.

2 is a block diagram of a depth map image analyzing unit 140 in a plurality of camera-based drone obstacle detection systems according to an embodiment of the present invention.

The depth map image analyzer 140 may include a depth map information analyzer 200 that analyzes the input depth map and classifies the area according to the depth map in the image.

The depth map image analyzer 140 may include an obstacle candidate analyzer 210 that classifies a region of danger of an obstacle based on the analyzed depth map classification region information.

The depth map image analyzer 140 may include a short range error extracting unit 220 classifying a region in which a matching error occurs because a parallax error occurs in the depth map.

In addition, the depth map image analyzer 140 registers a horizontal line detector 230 for detecting whether a horizontal line exists in the analyzed depth map classification region and a horizontal line detected by the horizontal line detector 230 for future horizontal obstacle determination. The horizontal obstacle candidate extractor 240 and the tilt phenomenon detection unit 250 may be included. In more detail, the near error extraction unit 220 may generate a parallax error event and register the error. In more detail, the horizontal obstacle candidate extractor 240 may determine and register a horizontal obstacle error event. In more detail, in the tilt phenomenon detection unit 250, error determination and registration for the drone tilt phenomenon may occur.

3 is a block diagram of the sub-camera control and recognition unit 150 in the multi-camera drone obstacle detection system according to an embodiment of the present invention.

The sub camera control and recognition unit 150 serves to control additional cameras installed between the stereo cameras. At this time, the sub-camera controlled by the sub-camera control and recognition unit 150 may play various roles to complement the stereo camera-based drone obstacle detection system.

The sub camera control and recognition unit 150 controls the operation of the sub camera when the result of analyzing the occurrence of the obstacle and the error in the depth map image analyzer 140 is registered as an event. In this case, examples of the event may include a parallax error, a drone tilt phenomenon, and registration of a road obstacle candidate.

In one embodiment of the present invention, the sub-camera control and recognition unit 150 operates when an event that a parallax error occurs is registered.

A parallax error is an error in which matching is not performed because the parallax is too large between the images taken by the left and right cameras when the short distance of the baseline between the stereo cameras is too far.

In this case, the sub camera may be located between the baselines of the pair of stereo cameras. At this time, in one embodiment of the present invention, the driving shaft to which the sub-camera and the driving motor are connected may be parallel to the straight line formed by the pair of stereo cameras.

In an embodiment of the present invention for correcting parallax errors, the sub-camera control and recognition unit 150 may perform stereo matching with an image obtained by the sub-camera after stereo matching between images input from the left and right cameras. In more detail, after performing stereo matching between the left and right cameras, the stereo matching may be performed again with the image obtained from one of the left and right cameras. Through this method, the present invention can recognize a near obstacle object and perform an operation of overcoming parallax errors.

In an embodiment of the present invention, the sub-camera control and recognition unit 150 operates when an event indicating that a tilt phenomenon occurs in the depth map image analyzer 140 is registered.

The sub-camera control and recognition unit 150 performs a function of solving a problem in which the drone is tilted and the camera is photographed downward due to a tilt phenomenon generated when the drone is flying at a high speed. In this case, the tilt phenomenon means a phenomenon in which obstacles are not recognized by shooting downward. Specifically, when the camera captures the lower part of the flight rather than the front of the flight, the area under the horizontal horizon occupies a large part of the acquired image. At this time, the image below the horizon makes it difficult to accurately extract the depth map, and there is a problem in that it is impossible to properly grasp the obstacle in front due to the downward shooting.

The sub-camera control and recognition unit 150 periodically analyzes the IMU information installed in the drone to determine whether the tilt occurs in the drone. At this time, the sub-camera control and recognition unit 150 compensates for the phenomenon that appears as a tilt phenomenon by changing the position of the sub-camera according to the tilt angle through the control of a physical device such as a motor when an event indicating that a tilt phenomenon has occurred is registered. Perform the action. A detailed tilt image complementary technique will be described later with reference to the accompanying drawings.

In an embodiment of the present invention, the sub-camera control and recognition unit 150 operates when an event indicating that a horizontal obstacle is found in the depth map image analyzer 140 is registered. That is, the sub-camera control and recognition unit 150 performs a function of determining a horizontal obstacle such as an electric wire that is difficult to detect with a general stereo image during flight of the drone.

When the event that the horizontal line is detected by the depth map image analyzing unit 140 occurs, the sub camera control and recognition unit 150 controls the sub camera to determine whether it is a horizontal obstacle. More specifically, when there is one horizontal line in the image, it may be determined as a horizon line. Therefore, there is no horizontal obstacle at this time. However, if there are two or more horizontal lines in the image, one or more of them may be a horizontal obstacle. In this case, the horizontal obstacle may be determined to generate a horizontal obstacle event.

More specifically, the sub-camera control and recognition unit 150 rotates and moves the sub-camera about the driving motor by using a physical device such as a motor when a horizontal line detection event occurs. At this time, in an embodiment of the present invention, the sub camera may be aligned in series with one of the stereo cameras. More specifically, the connecting shaft between the pair of stereo cameras and the driving shaft to which the driving motor and the sub camera are connected may form a vertical axis.

At this time, the sub-camera control and recognition unit 150 performs vertical stereo matching instead of general horizontal stereo matching and extracts a depth map thereof.

The sub-camera control and recognition unit 150 analyzes the extracted depth map to determine whether the horizontal line is a horizontal obstacle. In more detail, parallax does not occur in a horizontal line rather than a horizontal obstacle. On the other hand, in the case of a horizontal obstacle such as a power line within the detection range of the drone, a vertical and horizontal binocular parallax may be generated to confirm that the horizontal obstacle.

The sub-camera control and recognition unit 150 for implementing the functions includes a sub-camera control driver 300, a tilt information analyzer 310, a near depth value measurer 320, a tilt image corrector 330, and a vertical The stereo matching unit 340 and the horizontal obstacle extraction unit 350 may be configured.

In this case, the sub-camera control driver 300 may control the sub-camera for each event when a parallax error, a horizontal obstacle candidate registration, or a drone tilt event occurs.

In this case, when the parallax error event occurs, the near depth measurement unit 320 may perform a function of correcting the parallax error by performing stereo matching with one of the stereo cameras again using the sub camera.

At this time, the tilt image corrector 330 may receive the state information of the current drone from the tilt information analyzer 310 to check the tilt state of the drone.

In addition, when the tilt event occurs, the tilt image corrector 330 may adjust the position of the sub camera according to the tilt angle to correct the tilt image by adjusting the position of the camera with a physical device.

The horizontal obstacle extractor 350 may physically move the sub camera when the horizontal obstacle candidate event occurs. At this time, in one embodiment of the present invention, the sub-camera may be vertically aligned with one of the stereo cameras. In addition, the horizontal obstacle extractor 350 may measure a vertical depth value through the vertical stereo image matching unit 340 to extract a horizontal obstacle such as a power line that cannot be detected by horizontal stereo matching.

In the present invention, only two cameras are used to detect horizontal obstacles, but both a stereo camera and an additional camera may be used to increase the allowable range or accuracy of resources.

As described above, according to an embodiment of the present invention, an obstacle detection system based on an existing stereo camera is provided by using a pair of stereo cameras and one additional camera that are physically controllable in a drone with limitations on weight and HW resources. Several shortcomings can be compensated for.

4 shows an example of a camera device configuration of a multi-camera-based drone obstacle detection system according to an embodiment of the present invention. At this time, the camera device configuration of the drone obstacle detection system may be composed of a stereo camera and a sub camera. At this time, in one embodiment of the present invention, the sub camera may be located above the base line of the stereo camera.

4A is a diagram illustrating a driving method of a sub camera when a binocular parallax error occurs during an event. 4A is a view showing the configuration of an initial camera. At this time, the sub-camera of the image acquisition unit 120 may be set to the normal mode at the initial operation of the device. In this case, the sub camera set in the normal mode may be initially located between the baselines of the pair of stereo cameras. At this time, in one embodiment of the present invention, the sub-camera may be located in the middle of the pair of stereo cameras, but is not limited to the above-described embodiment. At this time, in one embodiment of the present invention, the driving shaft to which the sub-camera and the driving motor are connected may be parallel to the straight line formed by the pair of stereo cameras. More specifically, the baseline of the stereo camera and the drive axis of the sub camera may be configured in parallel. In this case, as shown in FIG. 4A, the sub-camera may be positioned between a pair of stereo cameras to correct binocular disparity that may occur because the baseline of the pair of stereo cameras is too far.

FIG. 4B is a diagram illustrating a driving method of a sub camera when a tilt phenomenon of an unmanned aerial vehicle and a horizontal obstacle candidate registration event occur during an event. Referring to FIG. 4B, when an event expected by a tilting phenomenon of a drone and a horizontal obstacle such as an electric wire occurs, the sub camera is configured to compensate for this by controlling the sub camera through a physical device such as a motor.

At this time, in one embodiment of the present invention, the driving shaft to which the sub-camera and the driving motor are connected may be positioned at a constant angle with a straight line formed by the pair of stereo cameras.

When a tilt event occurs, in one embodiment of the present invention, the position of the sub camera may be adjusted by a physical device according to the tilt angle. At this time, the angle formed by the driving axis of the sub camera and the baseline formed by the pair of stereo cameras may have a specific angle. In this case, the specific angle may be an angle to correct an error caused by the tilt phenomenon.

When the horizontal obstacle candidate event occurs, according to an embodiment of the present invention, the sub camera may be aligned in series with one of the stereo cameras. That is, the angle of the base line formed by the driving axis of the sub camera and the pair of stereo cameras may be 90 degrees. In more detail, the driving axis of the sub-camera and the base line formed by the pair of stereo cameras are perpendicular to each other to perform vertical stereo matching.

5 is a diagram illustrating a binocular parallax error and an error correction method according to an embodiment of the present invention.

More specifically, FIGS. 5A and 5B illustrate binocular parallax errors that may occur due to a far baseline arrangement of a stereo camera, and FIG. 5C illustrates an example for solving the problem using a sub camera.

5A is a diagram illustrating a far stereo image divided into a left image and a right image. In FIG. 5A, an image acquired through a stereo camera may acquire an image having different parallaxes according to a distance from a target object.

5B is a diagram illustrating a near stereo image divided into a left image and a right image, in which a binocular disparity error occurs. When the baseline between the stereo cameras is too far, as shown in FIG. 5B, when the object in the short distance is photographed, the parallax of the left and right stereo images is too large to recognize them as different objects and thus may not match.

5C is a diagram illustrating a left image and a right image in which binocular parallax error is corrected according to an embodiment of the present invention. In the present invention, when binocular disparity occurs as shown in FIG. 5B, an image captured by an additional sub-camera is matched with an existing stereo image to correct a near binocular disparity error. In this case, as shown in FIG. 5C, the sub camera may be disposed between the stereo cameras. At this time, in one embodiment of the present invention, the sub-camera may be located in the middle between a pair of stereo cameras, but is not limited to the above-described embodiment. Also, the sub camera may be disposed on the base line of the pair of stereo cameras. At this time, in one embodiment of the present invention, the driving shaft to which the sub-camera and the driving motor are connected may be parallel to a straight line formed by the pair of stereo cameras.

FIG. 6 is a diagram illustrating an example of a tilt phenomenon that occurs when a drone moves at a high speed according to an embodiment of the present invention.

FIG. 6A is a diagram illustrating a depth map obtained from a general front image, and FIG. 6B is a diagram illustrating a depth map obtained by tilting when the drone is traveling at high speed. As shown in FIG. 6B, when the area under the horizontal line occupies most of the image, the depth map may not be easily extracted due to the depth map extraction.

6C illustrates an example for supplementing a tilt image by using a sub camera. As shown in FIG. 6C, the sub camera performs stereo matching with an image captured by a conventional stereo camera. At this time, the sub-camera is controlled using a motor, and the sub camera is moved by an angle that can compensate for the tilt of the tilt phenomenon to take an image. An image obtained by controlling the sub camera may capture an upper angle of the image rather than a stereo image obtained at the beginning of the operation of the present invention.

At this time, the sub-camera extracts the depth value by extending the depth map of the upper region not captured by the stereo camera based on the depth map generated from the stereo matching. In more detail, the image of the stereo camera and the sub camera photographed on the upper side of the stereo camera will be present. The overlapping portions between the images may be identified, and the depth map of the upper region photographed by the sub-camera may be extended from the overlapping portions. At this time, the image of the stereo camera in which the region below the intended shooting region is captured by the tilt phenomenon can be extended to the upper region. This can be used to compensate for missing front obstacles caused by tilting and to compensate for errors in depth map measurements caused by ground imaging.

7 is a diagram illustrating an example of a method of determining a horizontal obstacle according to an embodiment of the present invention. In the case of a horizontal obstacle, even if a horizontal line is detected, it is necessary to determine whether the horizontal line is actually a horizontal obstacle. That is, the case such as the horizon does not correspond to the horizontal obstacle, but the case such as the electrical wire in front of the drone may be determined as the horizontal obstacle.

7A is a diagram illustrating a typical stereo camera. In this case, when there are thin horizontal obstacles such as the horizon and the electric wire, conventional stereo image matching cannot distinguish them. At this time, the sub camera may exist between the stereo cameras as set in the normal mode of the present invention. More specifically, when the horizontal binocular disparity is measured by comparing the left image and the right image of the stereo camera, the disparity difference does not occur in both the wire and the horizontal line. That is, it is not possible to determine whether the horizontal parallax in the image is an obstacle that actually hinders the progress of the drone. This is because only horizontal binocular disparity can be checked.

In order to solve this problem, when the horizontal obstacle candidate is found as illustrated in FIG. 7B, the sub camera is arranged in a straight line with one existing stereo camera using a physical device such as a motor. In more detail, the base line of the stereo camera and the drive shaft of the sub camera of the present invention may be perpendicular to each other.

Afterwards, horizontal obstacles such as electric wires are determined using vertical binocular disparity between the sub camera image and the stereo camera image. More specifically, vertical stereo matching is performed between the stereo camera and the sub camera. At this time, according to an embodiment of the present invention, the stereo camera performing vertical stereo matching with the sub camera may be a stereo camera forming one straight line with the sub camera. When vertical stereo matching is performed, vertical binocular disparity can be checked, and the vertical binocular disparity determines whether it is a horizontal obstacle.

Referring to FIG. 7B, when stereo matching between the image acquired from the sub camera and the right image is performed, it may be determined whether a parallax difference occurs to determine whether the obstacle is actually an obstacle to the progression of the drone. More specifically, when the horizontal line detected in the image is a horizon rather than a horizontal obstacle, the horizontal line does not generate a parallax when the image is compared between the sub camera and the stereo camera. On the other hand, when the horizontal line detected in the image is a horizontal obstacle such as an electrical wire within the detection distance of the drone, the horizontal line is bilateral parallax when the image is compared between the sub camera and the stereo camera. Therefore, horizontal lines in the image at this time may be classified as horizontal obstacles.

8 is a flowchart illustrating an obstacle detection method according to an embodiment of the present invention.

Referring to FIG. 8, first, an input image is obtained through a plurality of cameras mounted in a drone (S810). According to an embodiment of the present invention, when a plurality of images are input from the stereo camera, the images of the sub cameras may be simultaneously operated. According to an embodiment of the present invention, the image of the sub camera may be operated and input only when the above-described events occur.

When a plurality of stereo input images are obtained, the depth map generator 130 searches for the same points or regions of the left and right images (S820). In this case, when the same point or area of the left and right images is not searched, a parallax error event may be considered. As a stereo matching algorithm, a general stereo matching method such as a point or an area may be used, and an additional process such as calibration and image preprocessing may be required before performing stereo matching.

When stereo matching of the left and right images is completed, the depth map generator 130 generates a depth map based on each matching point or matching area (S830).

After generating the depth map of the entire image, it is necessary to perform the classification for the same areas in the generated depth map (S840). As a method of classifying regions in the depth map, a general region classification or a method of integrating adjacent regions may be used.

After the objects for the depth map are classified, operations for classifying the obstacle candidates according to whether the depth map has an error and the distance by analyzing the depth map are necessary (S850).

Through the depth map analysis, it may be determined whether a failure event has occurred (S860). In this case, examples of the event may include a parallax error, a drone tilt phenomenon, and registration of a road obstacle candidate.

If no event occurs, depth map analysis is performed until the event occurs. However, when an event occurs, the obstacle recognition unit 160 registers the obstacles identified in the operations and calculates the positions of the obstacles to inform the detection result S870.

Thereafter, the sub-camera is controlled according to the registered event to correct the error (S880). If the input image is not finished, the procedure of receiving the image and performing the depth map analysis is repeated (S890).

According to an embodiment of the present invention, when the sub camera control and recognition unit 150 recognizes that an event has occurred (S860), the sub camera control and recognition unit 150 may determine the sub camera according to the type of the event. The position and operation method can be controlled differently. The flow that operates according to the type of event in which the present invention occurs may be classified as shown in the following drawings. 9 is a flowchart illustrating a method for detecting an obstacle when a parallax error occurs, FIG. 10 is a method for detecting an obstacle when a tilt phenomenon occurs, and FIG. 11 is a flowchart illustrating a method for detecting an obstacle when a horizontal obstacle is detected. The contents to be analyzed for the operation of the sub camera control and recognition unit 150 according to the event type are as follows.

9 is a flowchart illustrating a method for detecting an obstacle when a parallax error occurs according to an embodiment of the present invention.

In order to confirm and correct the parallax error, first, a left and right image are input from the stereo camera (S910). In operation S920, whether a matching point or a matching area is searched for is obtained from the acquired left and right images. If no matching point or region is found in the left and right images, this is likely a parallax error. In one embodiment of the present invention, the calculation target of the matching point or the matching area may be a corner point or an edge point in the image. If a matching point or a matching area is not found in the acquired left and right images, it is registered as a parallax error event (S930). If a matching point or a matching area is found in the acquired left and right images, the depth map may be generated through stereo matching and the depth map may be analyzed.

When registered as a parallax error event, the sub camera may be used to correct the parallax error. After generating the depth map (S940), a parallax error region candidate is determined from the depth map (S950).

Then, the parallax error correction (S960) is performed on the parallax error area candidate area by using the sub camera.

More specifically, when it is confirmed that the binocular parallax error event is registered, the sub camera control and recognition unit 150 may arrange the sub camera between the stereo cameras. At this time, in one embodiment of the present invention, the sub-camera may be located in the middle between a pair of stereo cameras, but is not limited to the above-described embodiment. Also, the sub camera may be disposed on the base line of the pair of stereo cameras. At this time, in one embodiment of the present invention, the driving shaft to which the sub-camera and the driving motor are connected may be parallel to a straight line formed by the pair of stereo cameras.

At this time, the sub camera control and recognition unit 150 matches the image obtained from the sub camera with one of the left and right stereo cameras. Stereo matching with an image of a sub camera is used to correct a parallax error caused by a baseline distance between stereo cameras to determine a short-range obstacle. At this time, the selection of the left and right images of the stereo camera in which stereo matching is performed with the sub camera is irrelevant.

After that, it is determined whether the received image is terminated, and whether the event occurs through the depth map analysis is continued (S960).

10 is a flowchart illustrating a method for detecting an obstacle when a tilt phenomenon occurs in an embodiment of the present invention. Another depth map analysis required for sub camera control is the determination of the tilt phenomenon.

In order to check and correct the occurrence of the tilt phenomenon, first, an image is input from a device, and stereo matching is performed to generate a depth map. The generated depth map is analyzed (S1010).

Whether the tilt has occurred is determined by analyzing whether the first region is greater than or equal to the threshold in the image when the depth map is analyzed (S1020). In this case, the first area may mean a ground area, and may also mean an area other than the intended photographing area. In more detail, the drone moving at a high speed is inclined to move, and thus, the drone may photograph an area other than the intended photographing area, and may define it as a first area. That is, whether or not the tilt has occurred is determined based on whether the ground region occupies the threshold value or more in the image when the depth map is analyzed.

At this time, if it is determined that the first region is greater than or equal to the threshold value in the image, it may be determined that the tilt phenomenon has occurred (S1030). In more detail, as shown in FIG. 6B, when the target photographing area is smaller than the threshold, it may be determined that the drone is photographed downward due to the high speed movement.

At this time, the device of the present invention registers the current situation as a tilt phenomenon event (S1040). In this case, the threshold for determining the tilt phenomenon may be defined by a user or an operator or by information obtained from an IMU of an unmanned aerial vehicle.

When checking the tilt event of the image, the sub-camera control and recognition unit 150 determines how inclined the drone is currently flying based on the information received from the IMU of the drone. At this time, the inclination angle of the drone is grasped and the first angle is determined according to the grasped angle. The first angle refers to an angle formed by the screen to be photographed by the sub camera to compensate for tilting with the baseline of the stereo camera. In more detail, the sub camera should take a picture above the input screen of the stereo camera to compensate for the tilt, and the first sub-camera should be photographed to correct the tilt when compared to the input screen of the stereo camera. Can be defined in degrees. In this case, it may mean that the sub camera captures an image that is different from the input image of the stereo camera by a first angle.

At this time, the sub-camera control and recognition unit 150 controls the sub-camera by using a motor, and obtains an image by adjusting the sub-camera by an angle that can compensate for the tilt of the tilt phenomenon (S1050).

The image obtained by controlling the sub camera captures an upper angle than the stereo image obtained at the beginning of the operation of the present invention, and thus an image in which the tilt phenomenon is corrected may be obtained (S1060). In more detail, the sub-camera control and recognition unit 150 obtains an image obtained by capturing an upper angle from the sub-camera, and performs stereo matching on a region overlapped with the corresponding image and the left and right stereo images. On the other hand, the depth value outside the overlapped area is obtained by expanding the depth value obtained from the overlapped area to obtain the estimated depth value.

After that, it is determined whether the input image is terminated, and whether the event occurs through the depth map analysis is continued (S1070).

FIG. 11 is a flowchart illustrating an obstacle detecting method when detecting a horizontal obstacle according to an embodiment of the present invention. Horizontal obstructions are horizontal obstructions such as urban or ground wires and clotheslines, which are difficult to detect in a typical stereo system. Therefore, the present invention proposes a flow for detecting the horizontal obstacle through the sub camera control.

In order to detect the horizontal obstacle, the device first receives an image and performs stereo matching to generate a depth map. The generated depth map is analyzed (S1110).

During the depth map analysis, it is determined whether a horizontal line exists in the image (S1120). If there is a horizontal line in the image, it is registered as a horizontal obstacle candidate (S1130). More specifically, when there is one horizontal line in the image, it may be determined as a horizon line. Therefore, there is no horizontal obstacle at this time. However, if there are two or more horizontal lines in the image, one or more of them may be a horizontal obstacle. In this case, the horizontal obstacle may be determined to generate a horizontal obstacle event.

Therefore, at this time, since the horizontal line recognized in the image is not a real horizontal obstacle, such as a horizon, the sub-camera control and recognition unit 150 determines whether or not the actual horizontal obstacle.

When it is confirmed that the horizontal obstacle event occurs, the sub camera control and recognition unit 150 arranges the sub camera vertically with one of the stereo cameras using a motor or the like (S1140). In more detail, the base line of the stereo camera and the drive shaft of the sub camera of the present invention may be perpendicular to each other.

Thereafter, as shown in FIG. 7B, vertical stereo matching between the stereo camera and the sub camera is performed. At this time, according to an embodiment of the present invention, the stereo camera performing vertical stereo matching with the sub camera may be a stereo camera forming one straight line with the sub camera. When vertical stereo matching is performed, the vertical binocular disparity can be checked, and it is determined whether the horizontal obstacle is a vertical obstacle through the vertical binocular disparity (S1150).

More specifically, when the horizontal line detected in the image is a horizon rather than a horizontal obstacle, the horizontal line does not generate a parallax when the image is compared between the sub camera and the stereo camera. On the other hand, when the horizontal line detected in the image is a horizontal obstacle such as an electrical wire within the detection distance of the drone, the horizontal line is bilateral parallax when the image is compared between the sub camera and the stereo camera. Therefore, horizontal lines in the image at this time may be classified as horizontal obstacles.

Thereafter, it is determined whether the received image is terminated and the determination of whether an event occurs through the depth map analysis is continued (S1160).

Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments set forth below, but may be implemented in various forms, and only the present embodiments are intended to complete the disclosure of the present invention, and those skilled in the art to which the present invention pertains. It is provided to fully inform the person having the scope of the invention, which is defined only by the scope of the claims.

Hereinafter, the present invention will be described with reference to the drawings of a block diagram or a processing flowchart for explaining a drone obstacle detection system and a method thereof according to embodiments of the present invention. It will be appreciated that each block of the flowchart illustrations and combinations of flowchart illustrations may be performed by computer program instructions. Since these computer program instructions may be mounted on a processor of a general purpose computer, special purpose computer, or other programmable data processing equipment, those instructions executed through the processor of the computer or other programmable data processing equipment may be described in the flowchart block (s). It will create means to perform the functions.

Each block may also represent a module, segment or portion of code that includes one or more executable instructions for executing a specified logical function (s). It should also be noted that in some alternative implementations, the functions noted in the blocks may occur out of order.

100: drone detection system
120: image acquisition unit
130: depth map generator
140: depth map image analysis unit
150: sub camera control and recognition unit
160: obstacle recognition unit
200: depth map information analysis unit
210: obstacle candidate analysis unit
220: short-range error extraction unit
230: horizontal line detection unit
240: horizontal obstacle candidate extracting unit
250: tilt detection unit
300: sub camera control driver
310: tilt information analysis unit
320: near depth measurement unit
330: tilt image correction unit
340; Vertical Stereo Image Matching Unit
350: horizontal obstacle extraction unit

Claims (17)

In the obstacle detection method of the drone
Receiving an image from a pair of stereo cameras;
Generating a depth map from the input image; And
Analyzing the depth map to determine whether a failure event occurs;
Including but not limited to:
There is a sub camera between the pair of stereo cameras,
An image captured by the sub camera is changed based on the fault event,
Detecting an obstacle based on an image captured by the sub camera and an image of the pair of stereo cameras
Obstacle detection method of the drone, characterized in that. The method of claim 1
When comparing the images received from the pair of stereo cameras,
It is determined whether there is a matching point or area between the images,
When there is no matching point or area between the images
Obstacle detection method of the drone, characterized in that registered as the failure event. The method of claim 2
When the failure event is registered
The obstacle detection method of the drone, characterized in that the sub-camera is arranged so that the base line between the drive shaft of the sub-camera and the stereo camera in parallel. The method of claim 3
The image acquired by the sub-camera
The obstacle detection method of the drone, characterized in that the stereo matching with one of the left and right images obtained by the stereo camera. The method of claim 1
By analyzing the generated depth map,
It is determined whether the first region in the image is greater than or equal to the threshold,
If it is determined that the first region is greater than or equal to a threshold in the image,
Obstacle detection method of the drone, characterized in that registered as the failure event. The method of claim 5
When the failure event is registered
Grasp the tilt angle of the drone,
Determine a first angle according to the identified angle,
The sub-camera obstacle detection method of claim 1, characterized in that for taking an image that is different from the input image of the stereo camera by the first angle. The method of claim 6
To determine the overlapping range between the image obtained from the sub-camera and the image of the stereo camera,
After measuring the depth value of the overlapping range,
The obstacle detection method of the drone, characterized in that for extending the depth value of the overlapping range. The method of claim 1
When analyzing the generated depth map
Determine if there is a horizontal line in the image,
If it is determined that a horizontal line exists in the image,
If there are two or more horizontal lines in the image obstacle detection method of the drone, characterized in that registered as the failure event. The method of claim 8
When registered as the fault event
The obstacle detection method of the drone, characterized in that the sub-camera is disposed so that the base line between the driving axis of the sub-camera and the stereo camera perpendicular to each other. The method of claim 9
Vertical stereo matching is performed between the vertically disposed sub-camera and the stereo camera
And finally determining whether a horizontal line in the image is a real obstacle. In the obstacle detection device of the drone
An image acquisition unit which receives an image from a pair of stereo cameras;
A depth map generator for generating a depth map from the input image; And
A depth map image analyzer determining whether a failure event occurs by analyzing the depth map;
A sub camera controller;
Including but not limited to:
There is a sub camera between the pair of stereo cameras,
The sub camera controller
Change the image photographed by the sub camera based on the fault event;
Detecting an obstacle based on an image captured by the sub camera and an image of the pair of stereo cameras
Obstacle detection device of the drone. The method of claim 11
When comparing the images received from the pair of stereo cameras,
It is determined whether there is a matching point or area between the images,
The obstacle detection apparatus of the drone, characterized in that registered as the failure event when there is no matching point or region between the images. The method of claim 12
When the failure event is registered
The sub camera controller
The obstacle detection apparatus of the drone, characterized in that the sub-camera is arranged so that the baseline between the drive shaft of the sub-camera and the stereo camera is parallel. The method of claim 11
By analyzing the generated depth map,
It is determined whether the first region in the image is greater than or equal to the threshold,
If it is determined that the first region is greater than or equal to a threshold in the image,
Obstacle detection device of the drone, characterized in that registered as the failure event. The method of claim 14,
When the failure event is registered
Grasp the tilt angle of the drone,
Determine a first angle according to the identified angle,
The obstacle detection apparatus of the unmanned aerial vehicle, characterized in that the sub-camera to take an image that is different from the input image of the stereo camera by the first angle. The method of claim 11
When analyzing the generated depth map
Determine if there is a horizontal line in the image,
If it is determined that a horizontal line exists in the image,
If there are two or more horizontal lines in the image obstacle detection apparatus of the drone, characterized in that registered as the failure event. The method of claim 16
When registered as the fault event
The sub camera controller
The sub-camera obstacle detecting device of claim 1, wherein the sub-camera is arranged such that the base line between the driving shaft of the sub-camera and the stereo camera is perpendicular to each other.

Images