KR20220068606A - Automatic landing algorithm of drone considering partial images - Google Patents

Abstract

The present invention relates to an automatic landing algorithm of a drone considering a partial image and, more specifically, to an automatic landing algorithm of a drone considering a partial image, which may use an image filmed by an image sensor to generate learning data with respect to the entire and partial image of a landing pad before recognizing the landing pad by using a convolutional neural network (CNN) algorithm, estimate the central point to estimate the location of the landing pad, estimate the landing location by utilizing the location and altitude of the drone, the pixel value of the central point of the landing pad, the characteristic parameter of the image sensor and the like, and correct the estimated location using a Kalman filter before utilizing the estimated landing location in order to generate a speed command to perform landing, thereby inducing a precise landing.

Description

Automatic landing algorithm of drone considering partial image {AUTOMATIC LANDING ALGORITHM OF DRONE CONSIDERING PARTIAL IMAGES}

The present invention relates to an automatic landing algorithm for a drone in consideration of partial images, and in detail, using an image captured from an image sensor to recognize a landing pad using a Convolutional Neural Network (CNN) algorithm, and to estimate the center point of the landing pad After estimating the position, estimate the landing position using the position and altitude of the drone, the pixel value of the center point of the landing pad, and the characteristic parameters of the image sensor, correct the estimated position using the Kalman filter, and then use the estimated landing position. It relates to an automatic landing algorithm of a drone that considers partial images to induce precision landing by generating a speed command for landing performance.

Drones are flying vehicles that fly by remote control or autonomously fly along a designated route, and are used in various fields such as camera shooting, military use, transportation, security, and personal use. Drones are generally used as small unmanned aerial vehicles operated manually by the operator's radio operation signal, and manned drones are also on the trend of commercialization.

Logistics services using drones are faster and more accurate than existing logistics services, and the demand for drone logistics services is increasing due to advantages such as being able to provide even in hazardous areas. For drone logistics service, technologies such as communication, traffic management, obstacle detection/avoidance, and precision landing are required, and various studies are being conducted in this regard, and in particular, research on landing using images is being conducted.

As an example, a method and apparatus for controlling drone landing, which is Korean Patent Registration No. 10-2022695, is disclosed.

The method and apparatus for controlling the drone landing uses an ultrasonic sensor and a GPS sensor of a drone to recognize a target, detect a first position of the target, and use an optical sensor and an IR sensor of the drone to control the RGB of the target Acquire an image and an IR image, and use the recognizer learned to recognize the target on which the drone will land from the RGB images and the IR image. Detects a second position of a target that is a target, uses the first position and second position to generate a precise position of a target that is a landing target, and based on the precise position of the target that is a landing target and the position of the drone, Landing can be controlled.

In addition, the domestic registered patent No. 10-2079727, an automatic drone landing apparatus and method using vision recognition, can change and display the size of the landing point where the drone will land according to the altitude of the drone, and detect the drone by photographing the landing point. may land at the designated landing site.

However, the conventional drone landing using the image has a problem in that it cannot perform precise landing only by recognizing only the position of the target and landing. In addition, in the case of landing using images, if there are situations such as limiting the shooting area due to the angle of view of the image sensor as the altitude descends, or the drone moves away from the landing site due to disturbance, only a partial image of the landing pad rather than the entire image occurs. There is a problem in that the landing pad may not be recognized by being acquired.

Domestic Registered Patent No. 10-2022695 Domestic Registered Patent No. 10-2079727

The present invention is to solve the above problems, using an image captured from an image sensor to generate learning data including a partial image of the landing pad, and then using a CNN (Convolutional Neural Network) algorithm to recognize the landing pad and , after estimating the landing pad position by estimating the center point, then estimating the landing position using the drone’s position and altitude, the pixel value of the center point of the landing pad, and the characteristic parameters of the image sensor, and after correcting the estimated position using the Kalman filter The purpose of this is to provide an automatic landing algorithm of a drone that takes into account partial images that generate a speed command for landing performance using the estimated landing position to induce a precise landing.

Features of the present invention for achieving the above object,

an image information input step of receiving an image from an image sensor; Landing pad recognition and center point estimation step of estimating landing pad recognition and center point through learning with learning data generated using the entire image and partial image of the landing pad obtained in the flight test; an image information output step of outputting the estimated pixel value of the center point of the landing pad; a landing position estimation step of estimating the position from the center point of the landing pad using the position and altitude of the drone, the output center point pixel value, and the characteristic parameters of the image sensor, but correcting the position estimation error using a Kalman filter; and a landing guidance command generation step of generating a landing guidance command for landing performance by using the estimated landing position.

Here, the landing pad recognition and center point estimation step uses a convolutional neural network (CNN) algorithm, and in a situation where only a partial image of the landing pad can be obtained, the landing pad recognition and Estimate the center point.

Here, the landing position estimation step uses a Kalman filter based on a constant velocity model.

Here, in the landing guidance command generation step, a speed command is generated through proportional control by using the estimated landing pad position and the drone position error to perform automatic landing of the drone, but the position error in the north and east directions After calculating each, the error is converted to an error in the X and Y directions according to the aircraft coordinate system, and the U and V speed commands are generated through the proportional control below, and then the W descent speed for landing is used as a constant speed value. .

According to the automatic landing algorithm of the drone considering the partial image of the present invention configured as described above, after generating learning data including the partial image of the landing pad using the image captured from the image sensor, a CNN (Convolutional Neural Network) algorithm is used to recognize the landing pad, estimate the center point to estimate the landing pad position, After correcting the estimated position, the estimated landing position can be used to generate a speed command to perform the landing to induce a precise landing.

1 is an operation flowchart for explaining an automatic landing algorithm of a drone in consideration of a partial image according to the present invention.
2 is an exemplary image showing an entire image of a landing pad obtained from an image sensor by an automatic landing algorithm of a drone in consideration of a partial image according to the present invention.
3 is an exemplary image illustrating a partial image of a landing pad obtained from an image sensor by an automatic landing algorithm of a drone in consideration of a partial image according to the present invention.
4 is a photograph for explaining a state in which landing pad recognition and center point estimation are performed by an automatic landing algorithm of a drone in consideration of a partial image according to the present invention.
5 is a diagram illustrating a relative coordinate system of a drone, an image, and the ground in order to estimate a landing pad position according to an automatic landing algorithm of a drone in consideration of a partial image according to the present invention.
6 is a graph showing the actual landing pad position and the estimated landing pad position and the relative distance to the aircraft position according to the automatic landing algorithm of the drone in consideration of the partial image according to the present invention.
7 is a histogram of a relative distance error calculated through an estimated position according to an automatic landing algorithm of a drone in consideration of a partial image according to the present invention.
8 is a conceptual diagram illustrating the concept of a Kalman filter to which an automatic landing algorithm of a drone considering a partial image according to the present invention is applied.
9 is a block diagram showing the configuration of a controller to which an automatic landing algorithm of a drone in consideration of a partial image according to the present invention is applied.
10 is a flight test photograph of a flight test result using an automatic landing algorithm of a drone in consideration of a partial image according to the present invention.
11 is a graph illustrating a landing position estimation result of a flight test result using an automatic landing algorithm of a drone in consideration of a partial image according to the present invention.
12 is a landing performance trajectory of a flight test result using an automatic landing algorithm of a drone in consideration of a partial image according to the present invention.

Hereinafter, an automatic landing algorithm of a drone in consideration of a partial image according to the present invention will be described in detail with reference to the accompanying drawings.

In the following description of the present invention, if it is determined that a detailed description of a related well-known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, the terms to be described later are terms defined in consideration of functions in the present invention, which may vary according to intentions or customs of users and operators. Therefore, the definition should be made based on the content throughout this specification.

1 is an operation flowchart for explaining the automatic landing algorithm of a drone considering partial images according to the present invention, and FIG. 2 is a landing pad obtained from an image sensor by the automatic landing algorithm of a drone considering partial images according to the present invention. It is an exemplary image showing the entire image, and FIG. 3 is an exemplary image showing a partial image of the landing pad obtained from the image sensor by the automatic landing algorithm of the drone in consideration of the partial image according to the present invention, and FIG. 4 is a partial image according to the present invention It is a photograph for explaining the landing pad recognition and center point estimation by the automatic landing algorithm of the drone considering the image, and FIG. 5 is a picture for estimating the landing pad position according to the automatic landing algorithm of the drone considering the partial image according to the present invention It is a view showing the relative coordinate system of the drone, the image, and the ground for this purpose, and FIG. 6 is the actual landing pad position and the estimated landing pad position and the relative distance to the aircraft position according to the automatic landing algorithm of the drone in consideration of the partial image according to the present invention. 7 is a histogram of the relative distance error calculated through the estimated position according to the automatic landing algorithm of the drone considering the partial image according to the present invention, and FIG. 8 is a drone considering the partial image according to the present invention. It is a conceptual diagram showing the concept of a Kalman filter to which an automatic landing algorithm is applied, and FIG. 9 is a block diagram showing the configuration of a controller to which an automatic landing algorithm of a drone in consideration of a partial image according to the present invention is applied.

1 to 9 , the automatic landing algorithm of a drone considering a partial image according to the present invention includes an image information input step (S10), a landing pad recognition and center point estimation step (S20), and an image information output step (S30). ), and a landing position estimation step (S40) and a landing guidance command generation step (S50).

《Video information input step-S10》

First, an image is received from an image sensor (camera, CMOS image sensor, etc.) of the drone.

《Landing pad recognition and center point estimation step-S20》

Landing pad recognition and center point estimation are performed through learning with learning data generated using the landing pad image obtained from the flight test.

YOLOv2 is used among convolutional neural network (CNN) algorithms for landing pad recognition and center point estimation. , and through learning, landing pad recognition and center point estimation are performed as shown in FIG. 4 .

As shown in FIG. 5, the relative coordinate system of the drone, the image, and the ground is set, and the center of the relative coordinate system is changed to the position of the aircraft in real time. A rotation transformation matrix was used to transform the center point of the image coordinate system into the position of the ground coordinate system, and the actual landing pad position and the estimated relative distance between the landing pad position and the aircraft position are shown in FIG. 6 , when only a partial image of the landing pad is acquired from the image (yellow part in the drawing), it can be seen that the error is large.

《Video information output step-S30》

Then, the estimated pixel value of the center point of the landing pad is output.

《Landing position estimation step-S40》

,

는 각 방향의 상대거리,

,

는 각 방향의 상대속도를 의미한다.When only a part of the landing pad is visible in the image, a Kalman filter is used to correct the position estimation error, and the state variable, measured value, system model, and covariance matrix are set as in Equations 1 to 4. In Equation 1 and Equation 2

,

is the relative distance in each direction,

,

is the relative speed in each direction.

In the case of relative speed, the landing pad was a stationary object, so it was calculated using the drone speed, and the Kalman filter used a constant velocity model. In the case of the part where the landing pad was not recognized, the estimated value was set as the predicted value. When only the part of the landing pad was recognized, the position estimation performance was improved by setting it to reflect the relative speed more. 7 is a histogram of a relative distance error calculated through an estimated position, Table 1 below is RMSE (Root Mean Square Error), and FIG. 8 is a concept of a Kalman filter.

《Landing guidance command generation step-S50》

When the correction of the position estimation error is completed, a landing guidance command for landing is generated using the estimated landing position.

At this time, a speed command is generated by using the estimated position of the landing pad and the position error of the unmanned aerial vehicle to perform automatic landing of the drone. After calculating the position errors in the north and east directions, the corresponding errors are converted into errors in the X and Y directions according to the drone coordinate system, and U and V speed commands are generated through proportional control as shown in Equation 5. The descent speed for landing uses a constant speed value as shown in Equation (6). A controller configured for landing guidance may be represented as shown in FIG. 7 .

《Flight Test》

10 is a flight test photograph of a flight test result using the automatic landing algorithm of a drone considering a partial image according to the present invention, and FIG. 11 is a landing of a flight test result using the automatic landing algorithm of a drone taking a partial image according to the present invention It is a graph showing the position estimation result, and FIG. 12 is the landing performance trajectory of the flight test result using the automatic landing algorithm of the drone considering the partial image according to the present invention.

In the present invention, a flight test was performed to verify the implemented algorithm, and the drone used for the test used a gimbal to stabilize the image regardless of the drone's posture. As a Flight Control Computer (FCC), an open source-based Pixhawk was used. As MC (Mission Computer), ROS installed Jetson Xavier AGX was used to create and input control commands. FCC and MC communicate between modules through the Mavlink protocol, and during flight mode, input a speed command in offboard mode to induce landing. When entering the offboard mode, the algorithm according to the present invention is applied and operated.

After configuring the environment as above, a flight test was performed as shown in FIG. 10 . The automatic landing algorithm was performed at a position slightly off the center of the landing pad, and it was confirmed that landing was induced and performed in the direction of the center of the landing pad.

11 is a result showing the relative distance to the drone based on the actual position and the position estimated by the automatic landing algorithm according to the present invention, and FIG. 12 shows the automatic landing trajectory of the drone. 11 , it can be seen that the landing pad position is normally estimated with an error of up to 0.6 m in the north direction and up to 0.5 m in the east direction. Also, in the case of altitude, it can be confirmed that the landing is performed while descending without any abnormality according to a constant speed command from the automatic landing starting position. Although an error occurred in the landing position of the drone with the center position of the landing pad through FIG. 12 , it can be confirmed that the landing was performed stably.

The present invention may be variously modified and may take various forms, and in the detailed description of the invention, only specific embodiments thereof have been described. However, it is to be understood that the present invention is not limited to the particular form recited in the detailed description, but rather, it is to be understood to cover all modifications and equivalents and substitutions falling within the spirit and scope of the present invention as defined by the appended claims. should be

Claims (4)

an image information input step of receiving an image from an image sensor;
Landing pad recognition and center point estimation step of estimating landing pad recognition and center point through learning with learning data generated by using the entire image and partial image of the landing pad obtained in the flight test;
an image information output step of outputting the estimated pixel value of the center point of the landing pad;
A landing position estimation step of estimating the position from the center point of the landing pad using the position and altitude of the drone, the output center point pixel value, and the characteristic parameters of the image sensor, but correcting the position estimation error using a Kalman filter; and
Automatic landing algorithm of a drone considering partial images, characterized in that it comprises a landing guidance command generation step of generating a landing guidance command for landing performance using the estimated landing position. The method of claim 1,
The landing pad recognition and center point estimation step is,
Using a CNN (Convolutional Neural Network) algorithm, considering a situation where only partial images of the landing pad can be acquired, a partial image characterized by estimating the landing pad recognition and center point through learning with the generated learning data including the partial image The automatic landing algorithm of the considered drone. The method of claim 1,
The landing position estimation step includes:
An automatic landing algorithm of a drone considering partial images, characterized by using a constant velocity model-based Kalman filter. The method of claim 1,
In the step of generating the landing guidance command,
In order to automatically land the drone, the speed command is generated through proportional control using the estimated landing pad position and the drone position error. Automatic drone considering partial images, characterized by using a constant speed value for the W descent speed for landing, after converting it to an error in the X and Y directions, generating the U and V speed commands through the proportional control below. landing algorithm.

Images