WO2018124688A1

WO2018124688A1 - Drone control device for collision avoidance

Info

Publication number: WO2018124688A1
Application number: PCT/KR2017/015463
Authority: WO
Inventors: 이선구; 이충구
Original assignee: 이선구; 이충구
Priority date: 2016-12-26
Filing date: 2017-12-26
Publication date: 2018-07-05
Also published as: KR20180075111A; KR101895343B1

Abstract

A drone control device for collision avoidance, according to an embodiment, comprises: a driving unit for driving a rotor blade of a drone; a sensor unit including an omnidirectional camera for acquiring an omnidirectional image by photographing all directions of the drone with a single external omnidirectional lens; and a processor, which performs image processing on the omnidirectional image so as to detect an object positioned around the drone, measures the distance between the drone and the object detected on the basis of at least two omnidirectional images, and controls the driving unit such that the drone urgently avoids the object when the distance between the object and the drone is less than or equal to a predetermined distance.

Description

Collision Avoidance Drone Control

The present invention relates to a collision avoidance drone control device that provides a function of avoiding collision with surrounding objects using an omnidirectional camera system.

Recently, drones (unmanned drones) have been commercialized, and drones have been used in various fields such as camera photography.

These drones are small unmanned drones and are generally operated manually by receiving operator's operation signals wirelessly.

This drone operation method is inefficient in that the operator must be together, and also poses a danger in that an accident may occur due to the operator's mistake.

In order to complement this drone operation method, a drone control system that installs a distance sensor and a camera on the drone to detect objects around the drone and sends an alarm to a remote controller when an object with a risk of collision is detected during remote control, or around a drone An unmanned drone control system has been developed to control the flight of objects avoiding objects.

However, since drones can move in all directions, such a drone control system requires multiple distance sensors and multiple short viewing angle cameras to detect objects located in all directions of the drone. There is an aesthetic problem.

On the other hand, the utilization of the 360-degree camera (the omnidirectional camera) has recently attracted attention.

In detail, the omnidirectional camera technology has been applied to vehicle-related fields such as vehicle periphery monitoring, and content production fields for providing virtual reality contents, and research and development are being actively conducted.

In addition, recently, a technique for implementing a flight-wide omnidirectional photographing system by applying an omnidirectional camera to a drone has also been proposed.

In detail, Korean Patent Publication No. 10-2015-0025452 (July 12, 2005) is disclosed as a prior art document related to the present invention, and the prior art document discloses a flight-wide photographing system.

According to the prior art, each of the drones is photographed in the air by using a plurality of single viewing angle cameras, and then a single 360-degree omnidirectional video is generated so that a wide photographing range can be expressed as a single photograph, and difficult to access. It can be easily photographed at the position to provide a flying all-round shooting system that can produce a variety of videos.

In detail, the photographing unit of the flightable omnidirectional photographing system may be provided with six camera modules having a short viewing angle installed in the flying unit, and the six camera modules are disposed in front, rear, left, and right directions, respectively, Four shooting areas and two shooting areas can be taken simultaneously. And, the omnidirectional omnidirectional photographing system overlaps the common photographing area of the omnidirectional photographs to generate one 360 degree omnidirectional photograph.

Accordingly, the prior art invention requires a plurality of cameras installed in different directions to photograph the omnidirectional, and the image and the image boundary portions cause heterogeneity to the observer in the process of matching the images captured by the different cameras. There is a problem. That is, the omnidirectional photographing system currently mounted on the drone should use a plurality of cameras or a camera including a plurality of external lenses, and only transmits the photographed omnidirectional image to the user's terminal. I'm not doing it.

In addition, as described above, distortion occurs in the process of matching images taken by different cameras, and thus there is a problem that the matched image cannot be trusted.

The present invention is to solve the above-mentioned problem, by using a sensor unit for capturing the omnidirectional with a single lens to accurately detect the object around the drone, collision control drone control device to self-control the drone to avoid the detected object To provide.

Collision avoidance drone control apparatus according to an embodiment, the drive unit for driving the rotary blades of the drone; A sensor unit including at least one omnidirectional camera capturing omnidirectional images with a single external lens; And image the omnidirectional image to detect objects around the drone, and measure the distance between the detected object and the drone based on at least two omnidirectional images, and when the distance between the object and the drone is less than a predetermined distance, And a processor that controls the driver to emergency avoid the object.

In this case, the processor may divide the omnidirectional image at intervals along the arc direction, and detect the object around the drone by image processing the divided image.

In addition, the processor may segment the omnidirectional image such that the segmentation angle is gradually increased as it is spaced apart from the omnidirectional image region matching the moving direction of the drone.

The sensor unit may include a first omnidirectional camera disposed under the drone and a second omnidirectional camera disposed under the first omnidirectional camera. The first omnidirectional camera and the second omnidirectional camera may be stereo cameras.

The processor may further include an image preprocessor for preprocessing an omnidirectional image to obtain a stereo image, a disparity calculator for stereo matching the stereo image to obtain disparity information, and a stereo based on the disparity information. It may include a segmentation unit separating the background and the foreground of the image, and an object identification unit detecting at least one or more objects in the separated foreground.

Further, the processor calculates a first disparity from the first image of the first omnidirectional camera, calculates a second disparity from the second image of the second omnidirectional camera, and calculates the first and second disparities. The distance between the object and the drone can be calculated in inverse proportion to the sum of the parity.

In addition, the processor may correct the first disparity according to the distance and the direction in which the point on which the subject is projected from the first image is spaced based on the focus position, and the point at which the subject is projected on the second image may be adjusted to the focus position. The second disparity may be corrected according to the distance and the direction separated from each other, and the distance between the object and the drone may be calculated based on the corrected first and second disparities.

The processor also converts the first image of the first omnidirectional camera into a first rectangular panoramic image, converts the second image of the second omnidirectional camera into a second rectangular panoramic image, and converts the first disc from the first rectangular panoramic image. The parity may be calculated, the second disparity may be calculated from the second rectangular panoramic image, and the distance between the object and the drone may be calculated based on the first disparity and the second disparity.

The collision avoidance control apparatus according to the embodiment obtains the omnidirectional image by capturing the omnidirectional image of the drone with a single external omnidirectional lens, and processes the omnidirectional image to detect objects around the drone, based on at least two omnidirectional images. It measures the distance between the detected object and the drone, and when the distance between the object and the drone is less than the predetermined distance, the drone controls the propeller to emergency avoid the object, providing emergency collision avoidance function to safely drive the drone. Can be.

In addition, such a collision avoidance control device, by using only one, as many as two omnidirectional cameras, can monitor the drone all around at once, there is an advantage that the manufacturing cost is reduced because there is no need for multiple cameras or distance sensors, Since the entire omnidirectional image is captured by a single external omnidirectional lens, there is less distortion, and thus, the position of the object can be accurately detected.

Furthermore, the collision avoidance control apparatus according to the embodiment may analyze the omnidirectional image in consideration of the characteristics of the omnidirectional image photographed by a single external omnidirectional lens to more accurately detect the position of the object photographed in the omnidirectional image. .

1 is a side view of a drone equipped with a collision avoidance drone control device according to an exemplary embodiment of the present invention.

2 is a block diagram of a collision avoidance drone control device according to an embodiment of the present invention.

3 is a cross-sectional view of the omni-directional camera of the sensor unit according to an embodiment of the present invention.

4 is a cross-sectional view of the omni-directional camera of the sensor unit according to another embodiment of the present invention.

5 is a flowchart illustrating a process of providing an emergency object avoidance function by a collision avoidance drone control device according to an exemplary embodiment of the present invention.

6A is an example of a segmented omnidirectional image according to an embodiment of the present invention, and FIG. 6B is an example of a segmented omnidirectional image according to another embodiment of the present invention.

7 is an internal block diagram of a processor according to an embodiment of the present invention.

8 is a conceptual diagram illustrating a method of measuring a distance between an object and a drone by a processor according to an embodiment of the present invention.

9 is a flowchart illustrating a process of controlling the focus of the omnidirectional camera by the processor according to an embodiment of the present invention.

10 is a cross-sectional view of the omni-directional camera of the sensor unit according to another embodiment of the present invention.

11 is a side view of a drone equipped with a collision avoidance drone control device according to another embodiment of the present invention.

12 is an internal configuration diagram of the lower and upper omnidirectional cameras.

Figure 13 is a block diagram of a collision avoidance drone control device according to another embodiment of the present invention.

14 and 15 are cross-sectional views of the liquid lens.

FIG. 16 shows how a processor sets an optimal movement path of a drone according to an embodiment of the present invention.

Hereinafter, with reference to the drawings of the collision avoidance drone control apparatus according to an embodiment of the present invention will be described in detail. The following embodiments are provided as examples to sufficiently convey the spirit of the present invention to those skilled in the art. Therefore, the present invention is not limited to the embodiments described below and may be embodied in other forms. In the drawings, the size and thickness of the device may be exaggerated for convenience. Like numbers refer to like elements throughout the specification.

Advantages and features of the present invention, and methods for achieving them will be apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, only the embodiments are to make the disclosure of the present invention complete, and the general knowledge in the technical field to which the present invention belongs. It is provided to fully convey the scope of the invention to those skilled in the art, and the present invention is defined only by the scope of the claims. Like reference numerals refer to like elements throughout. In the drawings, the size and relative size of layers and regions may be exaggerated for clarity.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, “comprise” and / or “comprising” refers to a component, step, operation and / or element that is present in one or more other components, steps, operations and / or elements. Or does not exclude additions.

Referring to FIG. 1, a drone according to an embodiment may include a body 10 forming an outer shape of a drone, a propeller 20 rotatably disposed on the body 10, an object surrounding the drone, and detecting an object. It includes a collision avoidance drone control device for driving control of the propeller 20 to avoid.

In detail, the collision avoidance control apparatus according to the embodiment, by capturing the omnidirectional image of the drone with a single external lens to obtain a omnidirectional image, image processing the omnidirectional image to detect objects around the drone, based on at least two omnidirectional images The distance between the detected object and the drone, and when the distance between the object and the drone is less than a predetermined distance, the drone controls the propeller 20 to emergency avoid the object, it can provide an emergency collision avoidance function. .

Such a collision avoidance control device, by using only one, many two omnidirectional cameras 160, can monitor the drones at once, there is no need for multiple cameras or distance sensors, there is an advantage that the manufacturing cost is reduced Since the whole omnidirectional image is an image captured by a single external omnidirectional lens 200a, since there is little distortion, the position of the object can be accurately detected.

Furthermore, the collision avoidance control apparatus according to the embodiment analyzes the omnidirectional image in consideration of the characteristics of the omnidirectional image captured by the single external omnidirectional lens 200a to more accurately detect the position of the object photographed in the omnidirectional image. can do.

Hereinafter, each component of the collision avoidance drone controller will be described in more detail.

2, a collision avoidance control apparatus according to an embodiment includes an input unit 110, a communication unit 12, a driver 130, a memory 140, a power supply unit 150, a sensor unit 160, and a processor ( 170, and in particular, the sensor unit 160 may include at least one omnidirectional camera 200 and other sensors. However, the units of the collision avoidance control device shown in FIG. 2 may not be an essential unit for implementing the collision avoidance control device, and according to an embodiment, some of the units may not be included in the collision avoidance control device. Other units not shown may be further included in the collision avoidance control device.

First, the collision avoidance drone control apparatus may include an input unit 110 that detects a user input.

For example, the input unit 110 may execute execution input for turning on / off the power supply of the collision avoidance drone control device, or turn on / off the emergency drone avoidance function of the collision avoidance drone control device. Input can be detected, etc.

The input unit 110 may include a gesture input unit 110 for detecting a user gesture (for example, an optical sensor), a touch input unit 110 for detecting a touch (for example, a touch sensor, and a touch key). (touch key, push key, etc.), the user terminal 600, a microphone for detecting a voice input, a microphone, a remote remote controller, and at least one of a mobile terminal to detect a user input. .

In addition, the collision avoidance drone control device may include a communication unit 120 for wireless communication with the user's terminal.

In an exemplary embodiment, the communicator 120 may transmit the drone surrounding information obtained by analyzing the omnidirectional image and the omnidirectional image captured by the omnidirectional camera 200 to the terminal.

In addition, in an embodiment, the communication unit 120 may receive a remote drone control signal input by a user through the terminal, and the processor 170 controls the driver 130 according to the received remote drone control signal. In addition, it may provide a remote control function for the user to remote control the drone. However, if there is a risk of collision with the object even during remote control, the processor 170 may execute the collision avoidance control in preference to the remote control signal.

That is, the collision avoidance drone control apparatus may transmit the omnidirectional image so that the omnidirectional location of the drone can be confirmed as one image, thereby improving user convenience.

The communication unit 120 may exchange data with a terminal in a wireless manner. The wireless data communication method may be various data communication methods such as Bluetooth WiFi, Direct WiFi, APiX, LTE, or NFC.

Next, the collision avoidance drone control device may include a driving unit 130 for supplying power to the propeller 20 and controlling the propeller 20 rotational direction to control the movement direction / movement speed of the drone.

In the embodiment, the collision avoidance drone control device has been described as including the drive unit 130 directly, the drone is provided with a separate drive unit 130 and the collision avoidance drone control device is connected to the drive unit 130 as an interface By transmitting the control signal, an embodiment of controlling the driver 130 may be possible.

The driver 130 includes a power source driver 130, and the power source driver 130 may perform electronic control of a power source in the drone. For example, when the electric-based motor is a power source, the power source driver 130 may control the motor. Thereby, the rotation speed, torque, etc. of a motor can be controlled.

The steering driver 130 may perform electronic control of a steering apparatus in the drone. In detail, the steering driver 130 may change the propagation direction of the drone through the inclination of the propeller 20 and the power control of the different propellers 20.

In an embodiment, the processor 170 may detect a traveling direction to avoid the collision dangerous object when detecting the collision dangerous object, and control the steering driver 130 to advance the drone in the detected traveling direction.

In addition, the collision avoidance drone controller may include a memory 140, and the memory 140 includes various data for operation of the overall collision avoidance drone controller such as a program for processing or controlling the processor 170. Can be stored.

In addition, the memory 140 may store a plurality of application programs or applications that are driven by the collision avoidance drone controller, data for operating the collision avoidance drone controller, and instructions. The application program may be stored in the memory 140 and may be driven by the processor 170 to perform an operation (or function) of the collision avoidance drone controller.

In an embodiment, the memory 140 may store data for identifying an object included in the omnidirectional camera 200 image. For example, the memory 140 may store data for identifying what the object corresponds to by a predetermined algorithm when a predetermined object is detected in the image acquired through the camera.

For example, the memory 140 may store data for identifying what the object corresponds to by a predetermined algorithm when a predetermined object such as a tall building, a flying vehicle, or a bird is included in the image acquired by the camera. Can be.

In addition, the collision avoidance drone control apparatus includes a power supply unit 150, and the power supply unit 150 may supply power required for operation of each component under the control of the processor 170. The power supply unit 150 may be a battery inside the drone.

In addition, the collision avoidance drone control device may include a sensor unit 160 composed of at least one omnidirectional camera 200.

3 is a cross-sectional view of the omnidirectional camera 200 of the sensor unit 160 according to the embodiment of the present invention.

Referring to FIG. 3, the sensor unit 160 according to the embodiment may include one omnidirectional camera 200.

In detail, the omnidirectional camera 200 according to the embodiment includes a refraction part 240 formed to refract incident light, a horizontal part 260 formed horizontally at the end of the refraction part 240, and a horizontal part 260. And a reflection coating 250 which reflects incident light reflected from the inner reflection coating to provide incident light to the inner recess, and refracts the reflected light reflected through the reflection coating 250 to the lens array 290. Reflective coating of the incident light incident on the inner concave portion 210, the inner refractive portion 220 concave at the end of the inner concave portion 210, and the inner refractive portion 240, and incident from the refracting portion 240. An omnidirectional lens 200a including an inner reflection coating 230 for reflecting to 250 and an omnidirectional lens 200a are installed at the front, and a barrel having a lens array 290 formed therein ( 280).

Through the configuration of only one omnidirectional lens (200a) to the outside as described above it can provide an advantage capable of securing a 360-degree viewing angle.

In detail, the refracting part 240 is formed so that incident light is refracted, and the horizontal part 260 may be formed horizontally at the end of the refracting part 240. In this case, the reflection coating 250 may be formed on the horizontal portion 260 to reflect incident light reflected from the inner reflection coating 250 to provide incident light to the inner recess 210.

In addition, an inner recess 210 may be formed on the inner surface of the front recess lens 200a to refract incident light reflected through the reflective coating 250 to the lens array 290. An inner refractive portion 220 may be formed concave at the end of the 210.

In this case, the inner reflection coating 230 formed on the inner refractive portion 240 to reflect the incident light incident from the refracting portion 240 to the reflection coating 250 may be formed.

As described above, two refractive surfaces and two reflective coatings 250 may be configured.

In addition, since the omnidirectional lens 200a itself is not an aspherical surface, but is spherical, the processing is easy and the manufacturing process is simple, thereby providing convenience in manufacturing and manufacturing cost reduction.

That is, the refraction portion 240, the inner refraction portion 240 and the inner concave portion 210 is to have a spherical surface can solve the difficulty of machining, which is a problem of aspheric surface and at the same time can provide an omnidirectional photographing effect.

In addition, the omnidirectional lens 200a may be installed at the front of the barrel 280, and a lens array 290 may be formed inside the barrel 280. In addition, the inclination of the inner refractive portion 240 based on the virtual central axis 270 of the omnidirectional lens 200a may be steeper than that of the refracting portion 240. This is to form an inclination angle of the inner refractive portion 240 to be steeper than the inclination angle of the refracting portion 240 to satisfy the viewing range within the range of 35 to 85 degrees.

On the other hand, the reason for forming the inner recess 210 in the inner central portion of the omnidirectional lens 200a is to focus. In detail, when the incident light is incident on the lens array 290 by the inner recess 210, light may be collected and focused.

In the case of an ordinary optical lens, the image is formed according to the amount or reflection of light, and thus the design of the optical system can be manufactured in a complete body. That is, since the reflection of light generated through one overall shape is organic, it can be processed into a complete body from the initial design point.

4 is a cross-sectional view of the omnidirectional camera 200 of the sensor unit 160 according to another embodiment of the present invention.

Referring to FIG. 4, the sensor unit 160 according to the embodiment may include a first omnidirectional camera 200 and a second omnidirectional camera 201 disposed to be spaced apart from each other. In this case, the description of each of the first and second

omnidirectional cameras

200 and 201 overlaps with the description of the omnidirectional camera 200 described above, and thus a detailed description thereof will be omitted.

In detail, the first omnidirectional camera 200 may be disposed under the drone to photograph the omnidirectional view. At this time, the upper portion covered by the drone and the lower portion of the lower portion covered by the reflective coating 250 may not be photographed.

In addition, the second omnidirectional camera 201 may be disposed to extend to the reflective coating 250 of the first omnidirectional camera 200. In detail, one end of the barrel 281 of the second omnidirectional camera 201 may be disposed to overlap with the reflective coating 250 of the first omnidirectional camera 200.

That is, the first omnidirectional camera 200 and the second omnidirectional camera 201 are disposed to be spaced apart by a predetermined distance in the vertical direction (up and down direction), so that the first omnidirectional camera 200 and the second omnidirectional camera 201 are disposed. The camera may take two omnidirectional images having parallax with each other in the vertical direction.

In more detail, the sensor unit 160 according to the embodiment is configured as a omnidirectional stereo camera to position the first omnidirectional camera 200 and the second omnidirectional camera 201 at different positions so as to photograph the omnidirectional image with a parallax. Can be.

Therefore, the sensor unit 160 according to the embodiment may shoot the omnidirectional image of the drone with the two omnidirectional cameras 200, and may accurately measure the distance between the object and the drone photographed in the image.

A detailed description of a technique for detecting an object around a drone by using the image photographed by the omnidirectional camera 200 will be described later.

Next, the sensor unit 160 of the collision avoidance drone control device may further include at least one sensor of a motion sensor, an altitude sensor, a distance sensor, a balance sensor, or a vision sensor.

First, the motion sensor may be at least one magnetometer, accelerometer, gyroscope, acceleration gyro sensor. The motion information of the drone measured by the motion sensor may be used for drone control for collision avoidance.

In detail, the magnetometer is a sensor that functions as a compass, and measures magnetic north to obtain the drone's bearing information, and the processor combines the GPS position information, the magnetometer's bearing information, and the accelerometer's motion information to obtain the drone's motion information. Can be obtained.

In addition, the accelerometer may be a three-axis accelerometer, it is possible to measure the acceleration in the x, y, z axis direction to obtain the motion information, such as the speed and position of the drone, the inclination, the warp, the direction of the war. Such motion information may be utilized by the processor 170 to maintain the drone in a stable posture.

In addition, the gyroscope may acquire the inclination information of the drone by measuring the angular velocity in the x, y, z axis direction with a three-axis gyroscope.

The acceleration gyro sensor may be a sensor in which the three-axis acceleration sensor and the three-axis gyroscope sensor are combined, and further include a temperature sensor to acquire motion information.

Next, the altitude sensor may acquire the altitude of the drone by measuring atmospheric pressure, and may supplement the altitude information of the drone measured through GPS.

Next, the GPS sensor may measure the position, coordinates, and altitude of the drone through the satellite signal. It can be understood that such a GPS sensor is included in the communication unit.

Next, the apparatus may further include a distance sensor measuring a distance between the drone and the external object. For example, the distance sensor may measure the distance between the drone and the external object through an ultrasonic or laser output and return time. This distance information may be further used by the processor 170 to control the drone.

The apparatus may further include a vision sensor for capturing an external image and analyzing the photographed image pattern to determine the presence or absence of an obstacle. This vision sensor has a similar function to an omnidirectional camera, but can be driven when hovering at an altitude of 10m or less to check obstacles and to prevent collisions.

Next, the collision avoidance drone control device may include a processor 170 that controls the overall operation of each unit in the collision avoidance drone control device.

The processor 170 may include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, and microcontrollers. It may be implemented using at least one of a controller (micro-controllers), microprocessors (microprocessors), electrical units for performing other functions.

Hereinafter, a process of providing a collision emergency avoidance function by the collision avoidance drone control device including the above-described unit will be described in detail with reference to FIG. 5.

First, the sensor unit 160 may photograph an omnidirectional image around the drone. (S101)

In an embodiment, when the sensor unit 160 is configured as one omnidirectional camera 200, one omnidirectional camera 200 detects the light incident through the single external omnidirectional lens 200a to obtain an omnidirectional image around the drone. can do.

In another embodiment, when the sensor unit 160 includes an omnidirectional stereo camera, the two omnidirectional cameras 200 respectively detect light incident through the single external omnidirectional lens 200a to provide two omnidirectional images having parallax. Can be obtained.

In this case, the obtained omnidirectional image may be formed in a circular shape and may have a donut shape by the reflective coating 250.

Next, the processor 170 may divide the obtained omnidirectional image. (S102)

Although an embodiment of sensing an object by image processing the entire omnidirectional image is possible, the embodiment is for preventing an excessive load on the processor 170 when image processing the entire omnidirectional image of high-capacity data at once. In detail, the processor 170 may divide the omnidirectional image into a plurality of images and sequentially process the divided images to minimize the processing load occurring at the moment.

As the number of divisions of the omnidirectional image increases, the processor 170 may precisely detect an object disposed between the divided images, thereby improving resolution of sensing information around the drone.

In addition, the processor 170 may convert the sector-shaped split image into a square image, match the square images, and convert the omnidirectional image into a panoramic image.

For example, referring to FIG. 6A, the processor 170 may divide the omnidirectional image to have equal intervals in the arc direction.

The processor 170 may determine the priorities of the divided images, sequentially process the divided images according to the priorities, and analyze the objects included in the divided images.

In detail, the processor 170 may analyze the split image on the moving direction side of the drone as the highest priority, and analyze the split image adjacent to the moving direction side split image as the lower priority.

According to another embodiment, the processor 170 may divide the omnidirectional image into different sizes.

In detail, the processor 170 may designate the main surveillance region in the omnidirectional image, divide the main surveillance region into a plurality of images, improve resolution, and accurately detect an object in the primary surveillance region.

For example, referring to FIG. 6B, the processor 170 may move a region of the omnidirectional image that is matched to the direction in which the drone moves to have a small split angle θ1 and move away from the split image in the moving direction D side. The omnidirectional image may be divided to have a large division angle θ5.

In other words, the processor 170 divides the drone moving direction D side omnidirectional image area to increase resolution to accurately detect the moving direction D object, and divides the omnidirectional side image area farther from the moving direction. You can detect dangerous objects roughly.

In addition, when detecting a dangerous object approaching the drone through the omnidirectional image, the processor 170 may divide the area of the corresponding omnidirectional image into small portions to accurately detect the dangerous object.

Next, the processor 170 may extract an object photographed in the omnidirectional image. (S103)

Hereinafter, when the sensor unit 160 is composed of one omnidirectional camera 200, a method of sensing a photographed object will be described.

As a method in which the processor 170 extracts an object using one omnidirectional camera 200, there is a method of using only a change of pixel values between successive previous image frames and current image frames. When an object enters a surveillance target area where an object does not exist, a change occurs in the entire video frame photographed by the omnidirectional camera 200, which is an area where an object entering the surveillance target area is located on the entire video frame. This is indicated by the change in pixel value at. Accordingly, the processor 170 may compare the pixel values of the pixels constituting the previous image frame and the current image frame, respectively, and determine the region where the change in the pixel value is an object image.

As another method of extracting an object, a criterion for extracting an object from a current image frame may be set using a plurality of previous image frames. That is, the processor 170 generates a background image for extracting an object based on a plurality of previous image frames temporally preceding the current image frame, and generates pixels of the pixels constituting the background image and pixels of the current image frame. You can extract objects based on the difference in values. Since the region in which the pixel value does not change over the plurality of previous image frames is regarded as a background where no object exists, the processor 170 generates a background image using the same. If the area of the pixel value is changed between the background image and the current image frame as the object image, the object can be extracted more accurately than when comparing the previous image frame with the current image frame.

When the sensor unit 160 is configured as an omnidirectional stereo camera, the sensor 160 may detect the object more precisely. Hereinafter, a method of extracting an object photographed in the omnidirectional stereo image will be described in detail with reference to FIG. 7.

Referring to FIG. 7, as an example of an internal block diagram of the processor 170, the processor 170 may include an image preprocessor 410, a disparity calculator 420, an object detector 434, and an object tracking unit 440. , And the application unit 450 may be included. In FIG. 5 and the following description, an image is processed in the order of the image preprocessor 410, the disparity calculator 420, the object detector 434, the object tracking unit 440, and the application unit 450. It is not limited.

The image preprocessor 410 may receive two images from the omnidirectional stereo camera and perform preprocessing.

In detail, the image preprocessing unit 410 may include noise reduction, rectification, calibration, color enhancement, and color space conversion for two images. CSC), interpolation, and the omnidirectional camera 200 gain control. Accordingly, a sharper image may be obtained than the stereo image captured by the omnidirectional camera 200.

The disparity calculator 420 receives an image signal-processed by the image preprocessor 410, performs stereo matching on the two received images, and performs stereo matching. A disparity map may be obtained. That is, disparity information on the stereo image of the front of the drone may be obtained.

In this case, the stereo matching may be performed in units of pixels of stereo images or in units of predetermined blocks. The disparity map may refer to a map in which stereo parallax information of stereo images, that is, left and right images, is numerically represented.

The segmentation unit 432 may perform segmentation and clustering on at least one of the images based on the disparity information from the disparity calculator 420.

In detail, the segmentation unit 432 may separate the background and the foreground from at least one of the stereo images based on the disparity information.

For example, an area in which the disparity information is equal to or less than a predetermined value in the disparity map may be calculated in the background, and the corresponding part may be excluded. Thereby, the foreground can be relatively separated. As another example, an area in which the disparity information is greater than or equal to a predetermined value in the disparity map may be calculated in the foreground and a corresponding portion may be extracted. Thereby, the foreground can be separated.

As described above, by separating the foreground and the background based on the disparity information extracted based on the stereo image, the signal processing speed, the signal processing amount, and the like can be shortened in the subsequent object detection.

The processor 170 may calculate a positional relationship (eg, distance) between the detected object and the drone based on the disparity information extracted from the stereo image, and a detailed method will be described later.

Next, the object detector 434 may detect the object based on the image segment from the segmentation unit 432.

That is, the object detector 434 may detect an object with respect to at least one of the images based on the disparity information.

In detail, the object detector 434 may detect an object with respect to at least one of the images. For example, an object can be detected from the foreground separated by image segments.

Next, the object verification unit 436 may classify and verify the separated object.

To this end, the object verification unit 436 may include an identification method using a neural network, a support vector machine (SVM) method, a method of identifying by AdaBoost using a haar-like feature, or a histograms of oriented gradients (HOG). Techniques can be used.

The object checking unit 436 may check the objects by comparing the objects stored in the memory 140 with the detected objects.

For example, the object verification unit 436 can identify the surrounding aircraft, buildings, dangerous areas, and the like located near the drone.

The object tracking unit 440 may perform tracking on the identified object. For example, in order to sequentially identify the object in the obtained stereo images, calculate the motion or motion vector of the identified object, track the movement of the object, etc. based on the calculated motion or motion vector. Can be. As a result, it is possible to track surrounding vehicles, high-rise buildings, dangerous areas, etc., which are located around the drone.

Next, the application unit 450 may calculate a risk of collision or the like based on various objects located around the drone, for example, a surrounding vehicle, a high-rise building, a dangerous area, and the like. In addition, the possibility of collision with a vehicle or a building can be calculated.

The application unit 450 may output, as the drone surrounding information, a message or the like for informing the user of such information, based on the calculated risk, the likelihood of collision, the slip state, or the like. Alternatively, a control signal for attitude control or running control of the drone may be generated as the drone control information.

Meanwhile, the image preprocessor 410, the disparity calculator 420, the segmentation unit 432, the object detector 434, the object checker 436, the object tracking unit 440, and the application unit 450 are processors. It may be an internal configuration of the image processing unit in 170.

According to an exemplary embodiment, the processor 170 may include an image preprocessor 410, a disparity calculator 420, a segmentation unit 432, an object detector 434, an object checker 436, and an object tracking unit ( 440 and only a part of the application unit 450 may be included.

Next, the processor 170 may calculate a positional relationship between the detected object and the drone. This step may be executed simultaneously with the step of extracting the object from the omnidirectional image.

In detail, the processor 170 may compare the first omnidirectional image photographed by the first omnidirectional camera 200 and the second omnidirectional image photographed by the second omnidirectional camera 201, and may be separated from each other by an object. The separation distance can be calculated.

In more detail, the processor 170 may obtain the disparity information generated due to the parallax difference in the first omnidirectional image and the second omnidirectional image.

In addition, the processor 170 may correct the disparity information in consideration of the distortion generated by the omnidirectional image captured by the single external omnidirectional lens 200a. In detail, in the omnidirectional image, the object photographed near the center is photographed as being closer than the actual separation distance, and the object photographed near the outer periphery in the omnidirectional image is photographed as being farther than the actual separation distance.

Therefore, when the disparity information of the stereo image is not corrected, an error may occur in the separation distance.

The processor 170 according to the embodiment may accurately measure the distance between the object and the drone by correcting the disparity information of the stereo image by reflecting the distortion characteristic of the omnidirectional image.

Equation 1 is a formula for measuring a distance between a drone and an object based on a stereo image.

Where f = focal length, L = baseline between the first omnidirectional camera 200 and the second omnidirectional camera 201, dr = focal position of the first image plane from the point at which the subject is projected Distance to the first disparity, dl = distance from the point at which the subject is projected to the focal point of the second image plane (second disparity), a correction variable for correcting α = dr, β = dl Means the correction variable to be corrected.

In detail, referring to FIG. 8, the processor 170 may calculate a first disparity from the first image 200i of the first omnidirectional camera 200, and may calculate the second image (eg, the second omnidirectional camera 201). In operation 201i, the second disparity may be calculated, and the distance between the object and the drone may be calculated in inverse proportion to the sum of the first and second disparities.

Here, the first disparity means the distance from the point at which the subject is projected to the focal position of the plane of the first image 200i, and the second disparity is the second image 201i from the point at which the subject is projected. ) May mean a distance to a focal point of the plane.

As described above, since the first disparity and the second disparity contain distortions generated by the single external circular lens of the omni-directional camera 200, correction may be performed.

In detail, the processor 170 may correct the first disparity according to the distance and the direction in which the point on which the subject is projected in the first image 200i is spaced based on the focal position, and in the second image 201i. The second projection is corrected according to the distance from the point where the subject is projected based on the focus position and the separation direction, and the distance between the object and the drone based on the corrected first and second disparities. Can be calculated.

In more detail, the processor 170 may focus on the first image 200i because, if the position where the subject is projected in the first image 200i is close to the outer periphery of the image, the object may have been photographed as being farther than the actual position. The first disparity can be corrected by adding a correction variable α (negative) proportional to the distance between the position and the subject projection position.

In contrast, the processor 170 may determine that if the position where the subject is projected in the second image 201i is close to the center of the image, the object may have been photographed as being closer than the actual position. The second disparity can be corrected by adding a correction variable β (positive number) proportional to the distance between the subject projection positions.

Meanwhile, the processor 170 converts the first image 200i of the first omnidirectional camera 200 into a first rectangular panoramic image, and converts the second image 201i of the second omnidirectional camera 201 into a second rectangular image. Convert to a panoramic image, calculate a first disparity from the first rectangular panoramic image,

The second disparity may be calculated from the second rectangular panoramic image, and the distance between the object and the drone may be calculated based on the first disparity and the second disparity. That is, in the step of converting the omnidirectional image image into a quadrangular image, the distortion of the omnidirectional image may be corrected in advance.

As such, the processor 170 may accurately calculate the distance between the object and the drone in consideration of the distortion generated in the omnidirectional stereo image.

Next, the processor 170 may detect an object at risk of collision according to the positional relationship between the drone and the object. (S105)

In detail, when the distance between the drone and the object is within a predetermined distance, the processor 170 may determine the object as a collision risk object.

In this case, the processor 170 may determine the collision risk object by considering whether the rate of change of the distance between the drone and the object is negative or positive.

Next, when the collision risk object is detected, the processor 170 may calculate a collision avoidance direction and control to make an emergency turning in the collision avoidance direction of the drone through the driver 130. (S106)

In this case, the processor 170 may control the drone in preference to the remote control signal.

That is, the processor 170 may generate an emergency drone control signal in advance of the remote control signal, transmit the emergency drone control signal to the driver 130, and control the propeller 20, thereby further protecting the drone.

9 is a flowchart illustrating a process of controlling the focus of the omnidirectional camera 200 by the processor 170 according to an embodiment of the present invention.

On the other hand, as the drone continues to move, since the distance between the object and the drone changes, there is a need to control the focus of the omnidirectional camera 200.

Referring to FIG. 9, first, when a drone starts supporting and photographs an omnidirectional image, since the distance between the drone and the ground changes, it is necessary to control the focus of the omnidirectional camera 200. (S201, 202)

In detail, the processor 170 may store the speed at which the drone maintains in the memory 140, and automatically control the focus of the omnidirectional camera 200 according to the altitude at which the drone starts supporting and changes. (S203)

For example, when the drone supports at a predetermined speed, the processor 170 calculates the altitude of the drone in consideration of the time taken to start supporting and the predetermined speed, and photographs the ground side according to the altitude of the drone. It is possible to calculate the focus for the control of the omnidirectional camera 200 according to the calculated focus.

To this end, the omnidirectional camera 200 may further include a focus control unit for controlling the focus.

In detail, referring to FIG. 10, the omnidirectional camera 200 according to another embodiment may further include a focus lens for controlling focus and a focus controller for controlling the focus lens, together with the same configuration as the above-described embodiment. can do.

In detail, the omnidirectional camera 200 includes a fluid focus lens 320 disposed in the inner concave portion and including a different liquid, and a focus controller 310 including a current injection portion contracting or expanding the fluid focus lens 320. It may include.

FluidFocus lenses may include two types of liquids that are not mixed with each other. The two liquids may be liquids with different refractive indices, one with electricity and one without electricity. The focus control unit 310 injects current into one of the two liquids through the current injection unit, and the conductive liquid into which the current is injected changes in volume, so that the other liquids contract and expand like the cornea of the human eye. While the focus of the omnidirectional camera 200 can be controlled.

The fluid focus lens 320 is disposed in a shape corresponding to the inner concave portion 210 of the single external lens (the omnidirectional lens 200a) of the omnidirectional camera 200 to uniformly control the focus of the entire omnidirectional image. There are advantages to it.

Meanwhile, the processor 170 may detect the tracking object, calculate the distance to the tracking object, and then control the focus according to the calculated distance change. (S205)

Through such control, the processor 170 may accurately photograph a moving object, analyze the photographed image, and accurately calculate a distance from the photographed object, thereby further improving the performance of the emergency collision avoidance function. .

On the other hand, since the drone can move upward, there is a need to detect an object in the upper region of the drone.

11 is a side view of a drone equipped with a collision avoidance drone control device according to another exemplary embodiment of the present invention, and FIG. 12 is an internal configuration diagram of the lower and upper omnidirectional cameras.

11 and 12, a collision avoidance drone control device mounted on a drone according to another embodiment may include a lower omnidirectional camera 160a for detecting a lower peripheral area of a drone, and an upper peripheral area of a drone. It may include an upward omnidirectional camera 160b.

The upper omnidirectional camera 160b may have the same configuration as the omnidirectional camera of the above-described embodiment.

However, in the case of the image omnidirectional camera 160b, since it receives direct sunlight from the sun, it is preferable that the surface of the single external omnidirectional lens 200a is subjected to window tinting. That is, the upper omnidirectional camera 160b is the same as the omnidirectional camera described above, except that a window tinting process is applied to the surface of the single external omnidirectional lens 200a.

The processor detects the object photographed by the upper omnidirectional camera 160b, measures the distance between the object and the drone, and emergency avoidance control to move the drone downward as a dangerous object when the detected object is less than a predetermined distance. Can be performed.

In contrast, the processor detects the object photographed by the lower omnidirectional camera 160a, measures the distance between the object and the drone, and moves the drone upward as a dangerous object when the detected object is less than a predetermined distance. Emergency evasion control can be performed.

FIG. 13 is a block diagram of a collision avoidance drone control device according to still another embodiment of the present invention, and FIGS. 14 and 15 are cross-sectional views of a liquid lens.

13 to 15, a collision avoidance control apparatus according to another embodiment of the present invention includes an input unit 110, a communication unit 12, a driving unit 130, a memory 140, a power supply unit 150, The sensor unit 160 and the processor 170 may be included. In particular, the sensor unit 160 may include at least one omnidirectional camera 200 and a liquid lens 121.

The liquid lens 121 includes an oil layer 121c and an aqueous solution layer 121d stacked between the first protective glass 121a, the second protective glass 121b, the first and second

protective glasses

121a and 121b, The first insulating portion 121g disposed between the first electrode portion 121e and the second electrode portion 121f and the first and second electrode portions 121e and 121f disposed at the periphery thereof to insulate a voltage. And a second insulating portion 121h.

As the liquid lens 121 is supplied with power from the outside to the first and second electrode portions 121e and 121f, the radius of curvature and the thickness of the oil layer 121c are changed to pass through the liquid lens 121. The position of the focus of the light can be changed.

In more detail, when a predetermined voltage is applied to the first and second electrode portions 121e and 121f, the radius of curvature and the thickness of the oil layer 121c may increase. The focal length can be shortened by increasing the voltage.

The liquid lens 121 does not require a separate servo motor for the flow of the lens, thereby greatly reducing the manufacturing cost. In addition, precise refractive index change is possible.

In addition, the liquid lens 121 may be located between the lens array 290 and the omnidirectional lens 200a, may be located at the rear end of the lens array 290, any of the plurality of lenses in the lens array 290 It may be located between two lenses.

The processor 170 may control the power applied to the liquid lens 121 to vary the refractive index of the liquid lens 121. Therefore, by focusing from far to near, it is possible to check the approaching object precisely.When focusing at far distance, the angle of view becomes smaller in the direction of progress, and focuses about 500 times per second so that the object can be identified by image analysis. do.

Meanwhile, referring to FIG. 16, the collision avoidance drone control apparatus according to the embodiment may calculate an optimal movement path based on an image photographed through a camera and map information received through the communicator 120. The processor 170 may include a movement path setting unit 171 for calculating the optimum movement path.

In detail, referring to FIG. 16, obstacle 1 may be photographed in an area of an image that can be photographed through a camera in a drone. Therefore, the collision avoidance drone controller can set the optimal route to the destination while avoiding obstacle 1. However, since the area of the image that can be photographed by the drone has a limit, it may be difficult to calculate a moving path outside the photographing area.

In order to prepare for this, the collision avoidance drone control device may receive a map image matching the image through the communication unit 120. The collision avoidance drone controller may match the image with the received map image to determine a location where the current drone is located in the map image, and then set an optimal route to the destination while avoiding obstacles through the map image.

That is, since obstacle 2 cannot be photographed by the drone, as shown in FIG. 16, the obstacle 2 can be grasped in advance through the map information, and an optimum movement path can be set even for an area outside the photographing area.

On the other hand, the present invention can also be embodied as computer readable codes on a computer readable recording medium. Computer-readable recording media include all kinds of recording devices that store data that can be read by a computer system. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, and the like, and may also be implemented in the form of a carrier wave (for example, transmission over the Internet). Include. The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion. In addition, functional programs, codes, and code segments for implementing the present invention can be easily inferred by programmers in the art to which the present invention belongs.

In the detailed description of the present invention described above with reference to the preferred embodiment of the present invention, those skilled in the art or those skilled in the art having ordinary knowledge of the present invention described in the claims to be described later It will be understood that various modifications and variations can be made in the present invention without departing from the spirit and scope of the art. Therefore, the technical scope of the present invention should not be limited to the contents described in the detailed description of the specification but should be defined by the claims.

The present invention is a device mounted on the drone to control the movement to avoid the collision when the drone moves.

Claims

A driving unit for driving the rotating blades of the drone;

A sensor unit including an omnidirectional camera for capturing the omnidirectional image of the drone with a single external omnidirectional lens; And

Image the omnidirectional image to detect an object located near the drone,

Measuring a distance between the detected object and the drone based on at least two omnidirectional images,

When the distance between the object and the drone is less than a predetermined distance, the drone includes a processor for controlling the drive unit to emergency avoiding the object;

Drone control for collision avoidance.
The method of claim 1,

The processor,

The omnidirectional image is divided at intervals along the arc direction,

Image processing of the divided image to detect the object around the drone

Drone control for collision avoidance.
The method of claim 2,

The processor,

Dividing the omnidirectional image by dividing the region matching the moving direction of the drone in the omnidirectional image, and dividing the omnidirectional image so that the dividing angle gradually increases as it becomes farther from the region matching the moving direction.

Drone control for collision avoidance.
The method of claim 1,

The sensor unit,

A first omnidirectional camera disposed under the drone, and a second omnidirectional camera disposed under the first omnidirectional camera,

The first omnidirectional camera and the second omnidirectional camera constitute an omnidirectional stereo camera.

Drone control for collision avoidance.
The method of claim 4, wherein

The processor,

An image preprocessor for preprocessing the omnidirectional image to obtain a stereo image;

A disparity calculator for stereo matching the stereo image to obtain disparity information;

A segmentation unit that separates a background and a foreground of the stereo image based on the disparity information;

An object checker to detect at least one object in the separated foreground;

Drone control for collision avoidance.
The method of claim 6,

The processor,

Calculate a first disparity from the first image of the first omnidirectional camera,

Calculate a second disparity from a second image of the second omnidirectional camera,

Calculating a distance between the object and the drone in inverse proportion to the sum of the first and second disparities

Drone control for collision avoidance.
The method of claim 6,

The processor,

The first disparity is corrected according to the distance from which the subject is projected on the first image based on the focal position and the separation direction.

The second disparity is corrected according to a distance from the point where the subject is projected on the second image based on a focal position and a separation direction,

Calculating a distance between the object and the drone based on the corrected first disparity and the corrected second disparity

Drone control for collision avoidance.
The method of claim 1,

The sensor unit,

Further comprising at least one sensor of a motion sensor, an altitude sensor, a distance sensor, a balance sensor or a vision sensor

Drone control for collision avoidance.
The method of claim 8,

The distance sensor is an ultrasonic sensor, obtains distance information with an external object by measuring the reflected time after outputting the ultrasonic wave,

The processor,

The drone controls the driving unit to emergency avoid the object through the distance information.

Drone control for collision avoidance.
The method of claim 1,

Further comprising a communication unit for receiving map information including an image from an external server,

The processor,

Mapping the photographed omnidirectional image to the map information;

Calculating a movement route based on the mapped map information

Drone control for collision avoidance.